Hybrid Pluggable Processing Pipeline (HyP3): Personalized Cloud-Based Processing of SAR Data
Abstract
A problem often faced by Earth science researchers is the question of how to scale algorithms that were developed against few datasets to large volumes. This problem only gets bigger as we look to a future with larger and larger datasets becoming available. One significant hurdle can be the processing and storage resources required for such a task, not to mention the administration of those resources. As a processing environment, the cloud offers nearly unlimited potential for compute and storage, with limited administration required. The goal of the Hybrid Pluggable Processing Pipeline (HyP3) project was to demonstrate the utility of the Amazon cloud to process large amounts of data quickly and cost effectively.
The HyP3 system has been in use for about a year, and its main consumer is the SARVIEWS Hazard monitor for which HyP3 creates Interferometric and Terrain Corrected images of active volcanoes and earthquakes. Using HyP3 means that the latest images are available on the website within hours of data acquisition. As on demand processing has a huge range of applications, ASF would like to provide this capability to other organizations. Currently, this requires the creation of a new HyP3 account on the ASF infrastructure, and the cost of data processing falls directly on ASF. The HyP3 In a Box project turns the HyP3 system into a distributable Amazon Cloud template, allowing other users to host their own HyP3 system. This way, the user has full control over resource management and costs. For example, if a user has purchased a license for proprietary processing software, or develops their own processing software, this can be integrated internally into their own HyP3 In a Box system. This presentation will focus on the development techniques and enabling technologies that were used in turning HyP3 into a distributable and extensible cloud application. Hyp3 In A Box allows for users to processes as much data as they want as well as add their own processing algorithms to the pipeline if they so desire. This makes HyP3 powerful, flexible, easy to setup and while still allowing customization to experienced users.- Publication:
-
AGU Fall Meeting Abstracts
- Pub Date:
- December 2018
- Bibcode:
- 2018AGUFM.G41B0687W
- Keywords:
-
- 1240 Satellite geodesy: results;
- GEODESY AND GRAVITYDE: 1241 Satellite geodesy: technical issues;
- GEODESY AND GRAVITYDE: 1908 Cyberinfrastructure;
- INFORMATICSDE: 1932 High-performance computing;
- INFORMATICS