Alignment and Preprocessing¶

Once the data is made available as detailed in Data Ingestion, the next step is to ensure the data has been appropriately reshaped and aligned across data sources for consumption by the machine learning pipeline (or other analysis or simulation steps).

We'll be aggregating data across several years, using small versions of the Landsat images used in the Walker Lake example. See that topic for more work on calculating the difference between the water levels over time.

Recap: Loading data¶

We'll use intake to read in the small versions of the landsat_5 and landsat_8 data.

Do computations¶

We can do regular computations with thesexarray.DataArrays. For instance we can calculate the NDVI (vegetation index) for each of these image sets. Note that in landsat 5 the Red and NIR bands are stored in bands 3 and 4 respectively whereas in landsat 8 the Red and NIR are bands 4 and 5.

We can calculate the difference between these two years by subtracting one from the other.

Notice how pixelated that image looks. What is going on here? To figure it out, let's take a look at the shape of diff.

That's a lot smaller than our NDVI for each year. What is happening is that when we compute the difference on the data we only get values where there are values for each year in the same grid cell. Since the cells are on a different resolution this only happens once every so often. What we'd rather do is interpolate to the same grid and then do our computations on that.

Combine data from overlapping grids¶

These two sets of Landsat bands cover roughly the same area but were taken in 1988 and 2017. We have modified them to have different resolutions, different numbers of grid cells per image, and different x and y offsets. We can see that by printing the first x value for each year and seeing that they are not equivalent.

We can also do a quick check of resolution by subtracting the second x value from the first x value for each year.

Select region of interest¶

We'll define a central point in lat, lon convert that point to the coordinate reference system (CRS) of the data, and then use the area around the central point as the Region of Interest (ROI).

The first step is getting the CRS. This information is stored in the attributes of our original landsat data. Let's take a look at it:

So what we have is a proj4 string. We can convert that to a cartopy.crs object using the geoviews.util.proj_to_cartopy method.

Now with that cartopy.crs object, we can transform out lat, lon - which are in cartopy.crs.PlateCarree() to our new data crs.

Now we just need to define the area that we are interested in around this point. In this case we'll use a 30 km box around the center point.

Let's just check that bouding box captures the lake:

Regrid¶

We can use this region to define a new grid onto which we will interpolate our data.

We will use linear interpolation to calculate the values for each grid cell. This operation is not yet supported for dask arrays, so we'll first load in the data.

Now we can use the new coordinates defined above to interpolate from the input coordinates to the new grid. The options are nearest and linear with linear being selected by default.

Combining the data¶

Now that we have our data on the same grid we can combine our two years into one xarray object. We will treat the years as names and create an xarray.Dataset - a group of named xarray.DataArrays that share some of the same coordinates.

Vizualizing output¶

We can plot the arrays side by side:

Or we can calculate and plot the difference between the two years:

Side-note: Resampling¶

We can down-sample an xarray object by grouping the values into bins based on the desired resolution and taking the mean on each of those bins.

We'll use the left edge as the label for now.

See GeoViews docs for more information of resampling grids.

Next:¶

Now that the data have been aligned and preprocessed appropriately, we can then use them in subsequent steps in a workflow. Next we'll learn about Machine Learning.