ROVER: Robust Data Collection from the IRIS Data Management Center
Abstract
As access to high performance computing increases for researchers, so does the number of requests for very large volumes of data from the IRIS Data Management Center (DMC). Unfortunately, the web technologies used to provide bulk access to raw data do not inherently guarantee robust delivery. Collecting large data volumes, which often requires many hours or days to transfer, increases the chances of failures and an incomplete data set. Limitations in the employed standard data transfer mechanisms can keep failures from being detected by the user or the client software. To address this, the DMC created ROVER, a new tool that robustly downloads large volumes of data (terabytes) over extended periods of time.
The design of ROVER is simple, powerful, and addresses multiple use cases. At a high level, ROVER determines the availability of remotely stored data matching a users' request, compares the available data to a local, ROVER-maintained, repository, and downloads the discrepant data. These steps are repeated, up to a reasonable number of iterations, until all the available data have been collected. This method of data collection has the advantage of allowing new data to be added to a local repository simply by repeating or extending the request. ROVER has a number of advanced characteristics: Data are downloaded in parallel for faster data collection. A data index is created and used by ROVER and is also accessible to the user. A local repository can be augmented without downloading previously collected data. Data may be saved in either miniSEED or Adaptable Seismic Data Format (ASDF). Data can be retrieved from any data center offering standard FDSN services and the IRIS availability service. In this presentation we will describe the features of ROVER, anticipated use cases, and plans for its future development.- Publication:
-
AGU Fall Meeting Abstracts
- Pub Date:
- December 2019
- Bibcode:
- 2019AGUFM.S24C..08R
- Keywords:
-
- 7299 General or miscellaneous;
- SEISMOLOGY