Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels
Abstract
We establish a scale separation of Kolmogorov width type between subspaces of a given Banach space under the condition that a sequence of linear maps converges much faster on one of the subspaces. The general technique is then applied to show that reproducing kernel Hilbert spaces are poor $L^2$-approximators for the class of two-layer neural networks in high dimension, and that multi-layer networks with small path norm are poor approximators for certain Lipschitz functions, also in the $L^2$-topology.
- Publication:
-
arXiv e-prints
- Pub Date:
- May 2020
- DOI:
- 10.48550/arXiv.2005.10807
- arXiv:
- arXiv:2005.10807
- Bibcode:
- 2020arXiv200510807E
- Keywords:
-
- Mathematics - Functional Analysis;
- Computer Science - Machine Learning;
- Statistics - Machine Learning;
- 68T07;
- 41A30;
- 41A65;
- 46E15;
- 46E22