Earth Science Analysis-Ready Data 2.0

Earth Science Analysis-Ready Data (ARD) for research and applications has been produced for many years. The definition of ARD has changed over time and currently, Earth Science ARD is usually defined as data that is well-calibrated, has good geolocation (with terrain correction if appropriate), is in a well-documented self-describing format (e.g., HDF or NetCDF), has good metadata and quality flags, and is easily discoverable and accessible. In many cases this data is also gridded, and may be spatially and temporally aggregated. The data producer makes many choices when deciding various parameters for these products such as the grid specification, the temporal aggregation approach, etc. The goal is to select these parameters to make the products useful to the end-user but also to minimize the number and volume of products stored. For this discussion, these standard products can be considered ARD 1.0. Over time, the number of data sets has grown in volume, velocity and variety, so many users need to perform additional steps before they can use this Big Data in their analysis. These additional operations to make ARD 2.0 are in three broad categories: gridding, subsetting and reformatting. Many of these operations are available as post-processing options provided by the data distributor's Web interface, by tools at the user's facility, and/or increasingly as part of methods in Cloud analysis platforms. This talk will talk break down the details of these additional steps for generating ARD 2.0. Having this detailed list will enable data providers, tool developers and Cloud analysis platform builders to understand where gaps exist in the current set of tools required to fully make Earth Science data ready for analysis.