VarianceReduced Decentralized Stochastic Optimization with Accelerated Convergence
Abstract
This paper describes a novel algorithmic framework to minimize a finitesum of functions available over a network of nodes. The proposed framework, that we call \textbf{\texttt{GTVR}}, is stochastic and decentralized, and thus is particularly suitable for problems where largescale, potentially private data, cannot be collected or processed at a centralized server. The \textbf{\texttt{GTVR}} framework leads to a family of algorithms with two key ingredients: (i) \textit{local variance reduction}, that enables estimating the local batch gradients from arbitrarily drawn samples of local data; and, (ii) \textit{global gradient tracking}, which fuses the gradient information across the nodes. Naturally, combining different variance reduction and gradient tracking techniques result into different algorithms with valuable practical tradeoffs and design considerations. Our focus in this paper is on two instantiations of the \textbf{\texttt{GTVR}} framework, namely \textbf{\texttt{GTSAGA}} and \textbf{\texttt{GTSVRG}}, that, similar to their centralized counterparts (\textbf{\texttt{SAGA}} and \textbf{\texttt{SVRG}}), exhibit a compromise between space and time. We show that both \textbf{\texttt{GTSAGA}} and \textbf{\texttt{GTSVRG}} achieve accelerated linear convergence for smooth and strongly convex problems and further describe the regimes in which they achieve nonasymptotic, networkindependent convergence rates that are faster with respect to the existing decentralized schemes. Moreover, we show that both algorithms achieve a linear speedup in such regimes, in that, the total number of gradient computations required at each node is reduced by a factor of $1/n$, where $n$ is the number of nodes, compared to their centralized counterparts that process all data at a single node. Extensive simulations illustrate the convergence behavior of the corresponding algorithms.
 Publication:

arXiv eprints
 Pub Date:
 December 2019
 arXiv:
 arXiv:1912.04230
 Bibcode:
 2019arXiv191204230X
 Keywords:

 Mathematics  Optimization and Control;
 Electrical Engineering and Systems Science  Systems and Control