Imaging SKA-Scale Data on Cloud and Supercomputer Infrastructure Using Drops and DALiuGE

Just one of the Square Kilometre Array (SKA) Phase I science projects will produce data of the order of tens of terabytes per second. The SKA Phase I project will have stringent power constraints in order to limit the operational costs (Dewdney et al. 2013); it is a considerable challenge to manage, process and store such large datasets within these constraints.

The current state-of-the-art astronomy data processing systems are designed to handle data approximately two to three orders of magnitude smaller than the SKA Phase I. To tackle this challenge, we have developed the Data-Activated Flow Graph Engine (DALiuGE), as part of the prototyping effort for the Science Data Processor Consortium of the SKA Phase I design. DALiuGE aims to provide a distributed data management platform and a scalable pipeline execution environment to support continuous, time and power bounded, data-intensive processing for producing SKA science-ready products. In this paper, we provide a brief overview of DALiuGE.