Real-world Experiences with A General Purpose System for Server-side Analysis of Earth Science Data and its Application to Analysis and Visualization of Large Earth Science Data Sets
Abstract
Access to scientific data is getting easier. That's the good news. But, data collections are getting increasingly complex, both in the size of the data involved and it the complexity of the underlying data grid. Rather than providing access to data on highly sub-sampled regular grids, data are being made available on complex model-native grids. This has tremendous advantages for scientists wishing to make use of the data for initializing their own models or for exploring and analyzing the data using specialized domain-specific algorithms. However, certain important classes of users could best make use of sub-sampled data on regular grids and some times model-intercomparison can only be done effectively by interpolating each model grid onto a common representation. The expectations of these diverse needs places a burden on data providers which can be lessened by the user of automated tools such as the Ferret-THREDDS Data Server (F-TDS). We have talked in the past about interactive use of F-TDS server-side analysis by scientists and how software systems like the Live Access Server take advantage of these server-side analysis capabilities, but here we focus on the real-world experience we've gained recently as data providers expand and augment access to their data by defining transformations of complex model grids on to predetermined regular grids. These transformations are defined using precomputed mapping files and Ferret scripts and served via F-TDS. This makes the introduction of these new data sets very low effort for the data provider and very resource efficient since the actual computations are delayed until the data are requested, the computations are limited to the minimum region needed to fulfill the request and the intermediate requests are cached and reused whenever possible. Our talk will focus on these real-world experiences serving ROMS model data form the RENCI (RENaissance Computing Institute) and the federated analysis and visualization system that uses these techniques as part of the ESG-CET CMIP5 archive.
- Publication:
-
AGU Fall Meeting Abstracts
- Pub Date:
- December 2011
- Bibcode:
- 2011AGUFMIN41C..04S
- Keywords:
-
- 1916 INFORMATICS / Data and information discovery