Unified Optimal Analysis of the (Stochastic) Gradient Method
Abstract
In this note we give a simple proof for the convergence of stochastic gradient (SGD) methods on $\mu$-convex functions under a (milder than standard) $L$-smoothness assumption. We show that for carefully chosen stepsizes SGD converges after $T$ iterations as $O\left( LR^2 \exp \bigl[-\frac{\mu}{4L}T\bigr] + \frac{\sigma^2}{\mu T} \right)$ where $\sigma^2$ measures the variance in the stochastic noise. For deterministic gradient descent (GD) and SGD in the interpolation setting we have $\sigma^2 =0$ and we recover the exponential convergence rate. The bound matches with the best known iteration complexity of GD and SGD, up to constants.
- Publication:
-
arXiv e-prints
- Pub Date:
- July 2019
- DOI:
- 10.48550/arXiv.1907.04232
- arXiv:
- arXiv:1907.04232
- Bibcode:
- 2019arXiv190704232S
- Keywords:
-
- Computer Science - Machine Learning;
- Mathematics - Numerical Analysis;
- Mathematics - Optimization and Control;
- Statistics - Machine Learning
- E-Print:
- 11 pages, version 2 fixes typos and case distinction in the proof