Dynamics of stochastic gradient descent for twolayer neural networks in the teacherstudent setup
Abstract
Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data. We study this phenomenon by analysing the dynamics and the performance of overparameterised twolayer neural networks in the teacherstudent setup, where one network, the student, is trained on data generated by another network, called the teacher. We show how the dynamics of stochastic gradient descent (SGD) is captured by a set of differential equations and prove that this description is asymptotically exact in the limit of large inputs. Using this framework, we calculate the final generalisation error of student networks that have more parameters than their teachers. We find that the final generalisation error of the student increases with network size when training only the first layer, but stays constant or even decreases with size when training both layers. We show that these different behaviours have their root in the different solutions SGD finds for different activation functions. Our results indicate that achieving good generalisation in neural networks goes beyond the properties of SGD alone and depends on the interplay of at least the algorithm, the model architecture, and the data set. *This article is an updated version of: Goldt S, Advani M S, Saxe A M, Krzakala F and Zdeborova L 2019 Dynamics of stochastic gradient descent for twolayer neural networks in the teacherstudent setup Advances in Neural Information Processing Systems pp 698191.
 Publication:

Journal of Statistical Mechanics: Theory and Experiment
 Pub Date:
 December 2020
 DOI:
 10.1088/17425468/abc61e
 arXiv:
 arXiv:1906.08632
 Bibcode:
 2020JSMTE2020l4010G
 Keywords:

 machine learning;
 Statistics  Machine Learning;
 Condensed Matter  Disordered Systems and Neural Networks;
 Condensed Matter  Statistical Mechanics;
 Computer Science  Machine Learning
 EPrint:
 9 pages + references + supplemental material. Oral presentation at NeurIPS 2019. arXiv admin note: substantial text overlap with arXiv:1901.09085