Accelerating Climate Data Analysis and Visualization with Parallel Scripting

Climate models are continuing to increase both in their resolution and the number of variables used, resulting in multi-terabyte model outputs. This large volume of data overwhelms the series of processing steps used to derive climate averages and produce visualizations. Since many of the tasks in the post- processing sequence are independent, we have applied task-parallel scripting to speed up the post-processing. We have re-written portions of the complex shell script that processes output from the Community Atmosphere Model in Swift, a high-level implicitly-parallel scripting language that uses data dependencies to automatically parallelize a workflow. This has resulted in valuable speedups in model analysis for this heavily-used procedure. We describe the structure, usage, performance, and our experiences with the resulting script.