To Each Optimizer a Norm, To Each Norm its Generalization
Abstract
We study the implicit regularization of optimization methods for linear models interpolating the training data in the underparametrized and overparametrized regimes. Since it is difficult to determine whether an optimizer converges to solutions that minimize a known norm, we flip the problem and investigate what is the corresponding norm minimized by an interpolating solution. Using this reasoning, we prove that for overparameterized linear regression, projections onto linear spans can be used to move between different interpolating solutions. For underparameterized linear classification, we prove that for any linear classifier separating the data, there exists a family of quadratic norms ._P such that the classifier's direction is the same as that of the maximum Pmargin solution. For linear classification, we argue that analyzing convergence to the standard maximum l2margin is arbitrary and show that minimizing the norm induced by the data results in better generalization. Furthermore, for overparameterized linear classification, projections onto the dataspan enable us to use techniques from the underparameterized setting. On the empirical side, we propose techniques to bias optimizers towards better generalizing solutions, improving their test performance. We validate our theoretical results via synthetic experiments, and use the neural tangent kernel to handle nonlinear models.
 Publication:

arXiv eprints
 Pub Date:
 June 2020
 arXiv:
 arXiv:2006.06821
 Bibcode:
 2020arXiv200606821V
 Keywords:

 Computer Science  Machine Learning;
 Statistics  Machine Learning