Leveraging Serverless Cloud Architectures for Scientific Data Processing Systems

The Amazon Web Services cloud computing platform provides services and tools that allow the creation of inexpensive yet highly scalable serverless data processing pipelines. This presentation will give an overview of serverless cloud architectures in Amazon Web Services as well as practices and tools that supplement these systems. As part of this discussion, concepts such as "infrastructure as code", "serverless computing", and "continuous integration and delivery" will be described at a high level.

Implementing a science data processing system in the cloud is often performed using an approach that mirrors on-premise architectures. Typically, this involves a cluster and a head node server with which jobs are submitted to the cluster. These systems often require regular maintenance and may require an always on head node instance. However, the serverless approach is typically easier to maintain, and often more cost effective for some projects as well. This presentation will outline advantages and disadvantages of utilizing this approach. Diagrams will be provided illustrating workflow when utilizing the continuous deployment and architecture as code approaches, which pair well with serverless architectures. Diagrams of serverless data processing systems currently in development will also be provided.