On Statistical Efficiency in Learning
Abstract
A central issue of many statistical learning problems is to select an appropriate model from a set of candidate models. Large models tend to inflate the variance (or overfitting), while small models tend to cause biases (or underfitting) for a given fixed dataset. In this work, we address the critical challenge of model selection to strike a balance between model fitting and model complexity, thus gaining reliable predictive power. We consider the task of approaching the theoretical limit of statistical learning, meaning that the selected model has the predictive performance that is as good as the best possible model given a class of potentially misspecified candidate models. We propose a generalized notion of Takeuchi's information criterion and prove that the proposed method can asymptotically achieve the optimal outsample prediction loss under reasonable assumptions. It is the first proof of the asymptotic property of Takeuchi's information criterion to our best knowledge. Our proof applies to a wide variety of nonlinear models, loss functions, and high dimensionality (in the sense that the models' complexity can grow with sample size). The proposed method can be used as a computationally efficient surrogate for leaveoneout crossvalidation. Moreover, for modeling streaming data, we propose an online algorithm that sequentially expands the model complexity to enhance selection stability and reduce computation cost. Experimental studies show that the proposed method has desirable predictive power and significantly less computational cost than some popular methods.
 Publication:

arXiv eprints
 Pub Date:
 December 2020
 arXiv:
 arXiv:2012.13307
 Bibcode:
 2020arXiv201213307D
 Keywords:

 Mathematics  Statistics Theory;
 Computer Science  Information Theory;
 Computer Science  Machine Learning;
 Electrical Engineering and Systems Science  Signal Processing;
 Physics  Data Analysis;
 Statistics and Probability
 EPrint:
 to be published by the IEEE Transactions on Information Theory