Neural Tangent Kernel: Convergence and Generalization in Neural Networks
Abstract
At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinitewidth limit, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function $f_\theta$ (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinitewidth limit it converges to an explicit limiting kernel and it stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positivedefiniteness of the limiting NTK. We prove the positivedefiniteness of the limiting NTK when the data is supported on the sphere and the nonlinearity is nonpolynomial. We then focus on the setting of leastsquares regression and show that in the infinitewidth limit, the network function $f_\theta$ follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinitewidth limit.
 Publication:

arXiv eprints
 Pub Date:
 June 2018
 arXiv:
 arXiv:1806.07572
 Bibcode:
 2018arXiv180607572J
 Keywords:

 Computer Science  Machine Learning;
 Computer Science  Neural and Evolutionary Computing;
 Mathematics  Probability;
 Statistics  Machine Learning
 EPrint:
 In Advances in neural information processing systems (pp. 85718580) 2018