SparseInput Neural Networks for Highdimensional Nonparametric Regression and Classification
Abstract
Neural networks are usually not the tool of choice for nonparametric highdimensional problems where the number of input features is much larger than the number of observations. Though neural networks can approximate complex multivariate functions, they generally require a large number of training observations to obtain reasonable fits, unless one can learn the appropriate network structure. In this manuscript, we show that neural networks can be applied successfully to highdimensional settings if the true function falls in a low dimensional subspace, and proper regularization is used. We propose fitting a neural network with a sparse group lasso penalty on the firstlayer input weights. This results in a neural net that only uses a small subset of the original features. In addition, we characterize the statistical convergence of the penalized empirical risk minimizer to the optimal neural network: we show that the excess risk of this penalized estimator only grows with the logarithm of the number of input features; and we show that the weights of irrelevant features converge to zero. Via simulation studies and data analyses, we show that these sparseinput neural networks outperform existing nonparametric highdimensional estimation methods when the data has complex higherorder interactions.
 Publication:

arXiv eprints
 Pub Date:
 November 2017
 DOI:
 10.48550/arXiv.1711.07592
 arXiv:
 arXiv:1711.07592
 Bibcode:
 2017arXiv171107592F
 Keywords:

 Statistics  Methodology;
 Statistics  Machine Learning