Optimizing Data Compression for the Community Earth System Model
Abstract
Significant increases in computational power in recent years have enabled Earth system model simulations to run with finer resolutions and higher throughput, resulting in increasingly larger data volumes that often strain institutional storage resources and lead to storage restrictions. These storage limitations negatively impact science objectives by forcing scientists to run fewer or shorter simulations and/or output data less frequently.
Data compression offers a potential means of mitigating the data volume problem, and the National Center for Atmospheric Research (NCAR) has been investigating applying compression to data from the Community Earth System Model (CESM) for the last several years. Because lossless compression schemes (i.e., schemes that exactly reconstruct the data when decompressed) are relatively ineffective on floating-point simulation data, a lossy approach is required to meaningfully reduce data storage requirements. Lossy compression is able to achieve substantial storage reductions, but by its very definition is unable to exactly reproduce original values. Therefore, striking a balance between meaningfully reducing data volume and preserving the integrity of the simulation data is a critical and non-trivial task, particularly given the large and diverse set of climate variables. Our focus at NCAR is on gaining user acceptance via careful analysis and testing, and we describe the challenges and concerns when compressing climate data from CESM. We specifically discuss approaches and metrics for evaluating data loss due to compression, our current approach of optimizing compression for each variable, and our collaboration with compression algorithm development teams.- Publication:
-
AGU Fall Meeting Abstracts
- Pub Date:
- December 2019
- Bibcode:
- 2019AGUFM.A11A..08B
- Keywords:
-
- 3336 Numerical approximations and analyses;
- ATMOSPHERIC PROCESSES;
- 3337 Global climate models;
- ATMOSPHERIC PROCESSES