Catalyst Acceleration for GradientBased NonConvex Optimization
Abstract
We introduce a generic scheme to solve nonconvex optimization problems using gradientbased algorithms originally designed for minimizing convex functions. Even though these methods may originally require convexity to operate, the proposed approach allows one to use them on weakly convex objectives, which covers a large class of nonconvex functions typically appearing in machine learning and signal processing. In general, the scheme is guaranteed to produce a stationary point with a worstcase efficiency typical of firstorder methods, and when the objective turns out to be convex, it automatically accelerates in the sense of Nesterov and achieves nearoptimal convergence rate in function values. These properties are achieved without assuming any knowledge about the convexity of the objective, by automatically adapting to the unknown weak convexity constant. We conclude the paper by showing promising experimental results obtained by applying our approach to incremental algorithms such as SVRG and SAGA for sparse matrix factorization and for learning neural networks.
 Publication:

arXiv eprints
 Pub Date:
 March 2017
 arXiv:
 arXiv:1703.10993
 Bibcode:
 2017arXiv170310993P
 Keywords:

 Statistics  Machine Learning;
 Mathematics  Optimization and Control