GSKY - A high performance service for multi-petabyte sized datasets
Abstract
For researchers analysing, transforming, and integrating large geospatial datasets into their work, the traditional approach has been to either download a relevant part of the data and analyse these data subsets in an ad-hoc manner, or to invest significant work into batch processing large data and then storing and organising for further analysis. For many of our national data assets this has become infeasible due to the volume of storage space and the work needed to wrangle data at this scale. Recent developments in significant data repositories with integrated data processing infrastructure opens the door for new ways of processing data on demand.
The National Computational Infrastructure (NCI) has developed a highly distributed geospatial data server, called GSKY which provides a new capability for high performance data analysis. GSKY is currently being used in some national and international initiatives - providing fast access to programs and tools over the network, and allowing researchers to analyse NCI's multi-petabyte nationally significant research data collections: from satellite data products, climate and weather simulations, and rich geophysical data. GSKY supports on demand processing of data that allows interactive data exploration presented as an OGC standards-compliant interface via Web Map Services (WMS), Web Processing Services (WPS) or raw data arrays using Web Coverage Services (WCS). GSKY has functionality for specifying how ingested data should be aggregated, transformed and presented. It dynamically and efficiently distributes the requisite computations among computational nodes and thus provides a scalable analysis framework.- Publication:
-
AGU Fall Meeting Abstracts
- Pub Date:
- December 2019
- Bibcode:
- 2019AGUFMIN52A..08E
- Keywords:
-
- 0399 General or miscellaneous;
- ATMOSPHERIC COMPOSITION AND STRUCTURE;
- 3399 General or miscellaneous;
- ATMOSPHERIC PROCESSES;
- 1899 General or miscellaneous;
- HYDROLOGY;
- 1996 Web Services;
- INFORMATICS