Generalized massive optimal data compression
Abstract
In this paper, we provide a general procedure for optimally compressing N data down to n summary statistics, where n is equal to the number of parameters of interest. We show that compression to the score function - the gradient of the log-likelihood with respect to the parameters - yields n compressed statistics that are optimal in the sense that they preserve the Fisher information content of the data. Our method generalizes earlier work on linear Karhunen-Loéve compression for Gaussian data whilst recovering both lossless linear compression and quadratic estimation as special cases when they are optimal. We give a unified treatment that also includes the general non-Gaussian case as long as mild regularity conditions are satisfied, producing optimal non-linear summary statistics when appropriate. As a worked example, we derive explicitly the n optimal compressed statistics for Gaussian data in the general case where both the mean and covariance depend on the parameters.
- Publication:
-
Monthly Notices of the Royal Astronomical Society
- Pub Date:
- May 2018
- DOI:
- 10.1093/mnrasl/sly029
- arXiv:
- arXiv:1712.00012
- Bibcode:
- 2018MNRAS.476L..60A
- Keywords:
-
- methods: data analysis;
- Astrophysics - Cosmology and Nongalactic Astrophysics
- E-Print:
- 5 pages