Early alignment in twolayer networks training is a twoedged sword
Abstract
Training neural networks with first order optimisation methods is at the core of the empirical success of deep learning. The scale of initialisation is a crucial factor, as small initialisations are generally associated to a feature learning regime, for which gradient descent is implicitly biased towards simple solutions. This work provides a general and quantitative description of the early alignment phase, originally introduced by Maennel et al. (2018) . For small initialisation and one hidden ReLU layer networks, the early stage of the training dynamics leads to an alignment of the neurons towards key directions. This alignment induces a sparse representation of the network, which is directly related to the implicit bias of gradient flow at convergence. This sparsity inducing alignment however comes at the expense of difficulties in minimising the training objective: we also provide a simple data example for which overparameterised networks fail to converge towards global minima and only converge to a spurious stationary point instead.
 Publication:

arXiv eprints
 Pub Date:
 January 2024
 DOI:
 10.48550/arXiv.2401.10791
 arXiv:
 arXiv:2401.10791
 Bibcode:
 2024arXiv240110791B
 Keywords:

 Computer Science  Machine Learning;
 Statistics  Machine Learning