Failures of modeldependent generalization bounds for leastnorm interpolation
Abstract
We consider bounds on the generalization performance of the leastnorm linear regressor, in the overparameterized regime where it can interpolate the data. We describe a sense in which any generalization bound of a type that is commonly proved in statistical learning theory must sometimes be very loose when applied to analyze the leastnorm interpolant. In particular, for a variety of natural joint distributions on training examples, any valid generalization bound that depends only on the output of the learning algorithm, the number of training examples, and the confidence parameter, and that satisfies a mild condition (substantially weaker than monotonicity in sample size), must sometimes be very loose  it can be bounded below by a constant when the true excess risk goes to zero.
 Publication:

arXiv eprints
 Pub Date:
 October 2020
 arXiv:
 arXiv:2010.08479
 Bibcode:
 2020arXiv201008479B
 Keywords:

 Statistics  Machine Learning;
 Computer Science  Machine Learning;
 Mathematics  Statistics Theory
 EPrint:
 JMLR, 22(204):115, 2021