A Global Imaging Spectroscopy Data System: Big Data Access, Distribution and Utility Challenges and Solutions for Facilitating Science and Applications

The United States recently released the 2017 Earth Science Decadal Survey, and among their recommendations was a call to prioritize launching a spaceborne imaging spectrometer with global coverage with 16 days repeat for a longer-term mission than ever before (>2 years). The purpose is to enable new science and applications related to surface geology and biology. Because of the breadth and diversity of imaging spectroscopy, careful consideration to how the data is managed and the processing and distribution systems are designed will be crucial to enabling use of the data. A global imaging spectrometer mission would present at least three distinct but interrelated scalability challenges related to the end-to-end data system for science product processing (i.e., the Science Data System), and the archive: 1) the computational challenge of simply to storing and analyzing the data, 2) the distribution, optimization and standardization of Earth science and application algorithms for producing value-added information products, and 3) algorithmic consistency across regions and local scenes, such that global-scale maps of surface phenomena can be assessed and compared. This talk focuses on the first two of these three challenges - computational tractability and distribution - in the context of contemporary algorithms as well as future analyses that could benefit the consistency challenge. To achieve rapid processing, dynamic compute resources are necessary for standard or bulk reprocessing needs and to enable science that fully exploits the information embedded in these datasets. Cloud computing provides a scalable system that enables use of as many or as few nodes as necessary to process data. Building a data system on the cloud, however, has variable cost models depending on the data volume storage, frequency of access or download (i.e., "egress"), and the processing demand (speed/number of compute nodes). Thus, the objectives of this review are to examine the current computational costs of imaging spectroscopy processing algorithms from raw data (Level 0) to radiance (Level 1), surface reflectance (Level 2), to value-added information products (Level 3); and provide an overview of the relative strengths and weaknesses of different cloud architectures for processing data from a global imaging spectrometer.