Randomized Kaczmarz with geometrically smoothed momentum
Abstract
This paper studies the effect of adding geometrically smoothed momentum to the randomized Kaczmarz algorithm, which is an instance of stochastic gradient descent on a linear least squares loss function. We prove a result about the expected error in the direction of singular vectors of the matrix defining the least squares loss. We present several numerical examples illustrating the utility of our result and pose several questions.
 Publication:

arXiv eprints
 Pub Date:
 January 2024
 DOI:
 10.48550/arXiv.2401.09415
 arXiv:
 arXiv:2401.09415
 Bibcode:
 2024arXiv240109415A
 Keywords:

 Mathematics  Numerical Analysis;
 Mathematics  Probability;
 Statistics  Machine Learning
 EPrint:
 21 pages, 9 figures