On lower bounds for the biasvariance tradeoff
Abstract
It is a common phenomenon that for highdimensional and nonparametric statistical models, rateoptimal estimators balance squared bias and variance. Although this balancing is widely observed, little is known whether methods exist that could avoid the tradeoff between bias and variance. We propose a general strategy to obtain lower bounds on the variance of any estimator with bias smaller than a prespecified bound. This shows to which extent the biasvariance tradeoff is unavoidable and allows to quantify the loss of performance for methods that do not obey it. The approach is based on a number of abstract lower bounds for the variance involving the change of expectation with respect to different probability measures as well as information measures such as the KullbackLeibler or chisquaredivergence. Some of these inequalities rely on a new concept of information matrices. In a second part of the article, the abstract lower bounds are applied to several statistical models including the Gaussian white noise model, a boundary estimation problem, the Gaussian sequence model and the highdimensional linear regression model. For these specific statistical applications, different types of biasvariance tradeoffs occur that vary considerably in their strength. For the tradeoff between integrated squared bias and integrated variance in the Gaussian white noise model, we combine the general strategy for lower bounds with a reduction technique. This allows us to link the original problem to the biasvariance tradeoff for estimators with additional symmetry properties in a simpler statistical model. In the Gaussian sequence model, different phase transitions of the biasvariance tradeoff occur. Although there is a nontrivial interplay between bias and variance, the rate of the squared bias and the variance do not have to be balanced in order to achieve the minimax estimation rate.
 Publication:

arXiv eprints
 Pub Date:
 May 2020
 arXiv:
 arXiv:2006.00278
 Bibcode:
 2020arXiv200600278D
 Keywords:

 Mathematics  Statistics Theory;
 Statistics  Machine Learning;
 62G05