A Compute Perspective: Delivering Decision Support Products in 24 Hours from the Airborne Snow Observatory

The JPL Airborne Snow Observatory (ASO) must process 100s of GB of raw data to 100s of Terabytes of derived data in 24 hour Near Real Time (NRT) latency in a geographically distributed mobile compute and data-intensive processing setting. ASO provides meaningful information to water resource managers in the Western US letting them know how much water to maintain; or release, and what the prospectus of the current snow season is in the Sierra Nevadas. Providing decision support products processed from airborne data in a 24 hour timeframe is an emergent field and required the team to develop a novel solution as this process is typically done over months. We've constructed a system that combines Apache OODT; with Apache Tika; with the Interactive Data Analysis (IDL)/ENVI programming environment to rapidly and unobtrusively generate, distribute and archive ASO data as soon as the plane lands near Mammoth Lakes, CA. Our system is flexible, underwent several redeployments and reconfigurations, and delivered this critical information to stakeholders during the recent "Snow On" campaign March 2013 - June 2013. This talk will take you through a day in the life of the compute team from data acquisition, delivery, processing, and dissemination. Within this context, we will discuss the architecture of ASO; the open source software we used; the data we stored; and how it was delivered to its users. Moreover we will discuss the logistics, system engineering, and staffing that went into the developing, deployment, and operation of the mobile compute system.