Lossless Transformations and Excess Risk Bounds in Statistical Inference
Abstract
We study the excess minimum risk in statistical inference, defined as the difference between the minimum expected loss when estimating a random variable from an observed feature vector and the minimum expected loss when estimating the same random variable from a transformation (statistic) of the feature vector. After characterizing lossless transformations, i.e., transformations for which the excess risk is zero for all loss functions, we construct a partitioning test statistic for the hypothesis that a given transformation is lossless, and we show that for i.i.d. data the test is strongly consistent. More generally, we develop information-theoretic upper bounds on the excess risk that uniformly hold over fairly general classes of loss functions. Based on these bounds, we introduce the notion of a δ-lossless transformation and give sufficient conditions for a given transformation to be universally δ-lossless. Applications to classification, nonparametric regression, portfolio strategies, information bottlenecks, and deep learning are also surveyed.
- Publication:
-
Entropy
- Pub Date:
- September 2023
- DOI:
- 10.3390/e25101394
- arXiv:
- arXiv:2307.16735
- Bibcode:
- 2023Entrp..25.1394G
- Keywords:
-
- statistical inference with loss;
- strongly consistent test;
- information-theoretic bounds;
- classification;
- regression;
- portfolio selection;
- information bottleneck;
- deep learning;
- Computer Science - Information Theory;
- Computer Science - Machine Learning;
- Mathematics - Statistics Theory;
- Statistics - Machine Learning;
- 62C05;
- 62G10;
- 68P30;
- 94A17;
- G.3;
- I.5;
- H.1.1
- E-Print:
- to appear in Entropy