The Best of Both Worlds: Developing a Hybrid Data System for the ASF DAAC

The Alaska Satellite Facility (ASF) at the University of Alaska Fairbanks hosts the NASA Distributed Active Archive Center (DAAC) specializing in synthetic aperture radar (SAR). Historically, the ASF DAAC has hosted hardware on-premises and developed DAAC-specific software to operate, manage, and maintain the DAAC data system. In the past year, ASF DAAC has been moving many of the standard DAAC operations into the Amazon Web Services (AWS) cloud. This includes data ingest, basic pre-processing, archiving, and distribution within the AWS environment. While the cloud offers nearly unbounded capacity for expansion and a great host of services, there also can be unexpected and unplanned costs for such. Additionally, these costs can be difficult to forecast even with historic data usage patterns and models for future usage. In an effort to maximize the effectiveness of the DAAC data system, while still managing and accurately forecasting costs, ASF DAAC has developed a hybrid, cloud and on-premises, data system. The goal of this project is to make extensive use of the AWS cloud, and when appropriate, utilize on-premises resources to help constrain costs. This hybrid system attempts to mimic, on premises, a cloud environment using Kubernetes container orchestration in order that software can be run in either location with little change. Combined with hybrid data storage architecture, the new data system makes use of the great capacity of the cloud while maintaining an on-premises options. This presentation will describe the development of the hybrid data system, including the micro-services architecture and design, the container orchestration, and hybrid storage. Additional we will highlight the lessons learned through the development process, cost forecasting for current and future SAR-mission operations, and provide a discussion of the pros and cons of hybrid architectures versus all-cloud deployments. This development effort has led to a system that is capable and flexible for the future while allowing ASF DAAC to continue supporting the SAR community with the highest level of services.