Avoiding Synchronization in FirstOrder Methods for Sparse Convex Optimization
Abstract
Parallel computing has played an important role in speeding up convex optimization methods for big data analytics and largescale machine learning (ML). However, the scalability of these optimization methods is inhibited by the cost of communicating and synchronizing processors in a parallel setting. Iterative ML methods are particularly sensitive to communication cost since they often require communication every iteration. In this work, we extend wellknown techniques from CommunicationAvoiding Krylov subspace methods to firstorder, block coordinate descent methods for Support Vector Machines and Proximal LeastSquares problems. Our SynchronizationAvoiding (SA) variants reduce the latency cost by a tunable factor of $s$ at the expense of a factor of $s$ increase in flops and bandwidth costs. We show that the SAvariants are numerically stable and can attain large speedups of up to $5.1\times$ on a Cray XC30 supercomputer.
 Publication:

arXiv eprints
 Pub Date:
 December 2017
 arXiv:
 arXiv:1712.06047
 Bibcode:
 2017arXiv171206047D
 Keywords:

 Computer Science  Distributed;
 Parallel;
 and Cluster Computing;
 Computer Science  Machine Learning;
 Mathematics  Optimization and Control;
 Statistics  Machine Learning;
 68W10;
 90C25;
 G.1.6