Dropout Regularization Versus $\ell_2$-Penalization in the Linear Model
Abstract
We investigate the statistical behavior of gradient descent iterates with dropout in the linear regression model. In particular, non-asymptotic bounds for the convergence of expectations and covariance matrices of the iterates are derived. The results shed more light on the widely cited connection between dropout and l2-regularization in the linear model. We indicate a more subtle relationship, owing to interactions between the gradient descent dynamics and the additional randomness induced by dropout. Further, we study a simplified variant of dropout which does not have a regularizing effect and converges to the least squares estimator
- Publication:
-
arXiv e-prints
- Pub Date:
- June 2023
- DOI:
- 10.48550/arXiv.2306.10529
- arXiv:
- arXiv:2306.10529
- Bibcode:
- 2023arXiv230610529C
- Keywords:
-
- Mathematics - Statistics Theory;
- Statistics - Machine Learning
- E-Print:
- 52 pages, 2 figures