Global Convergence of Subgradient Method for Robust Matrix Recovery: Small Initialization, Noisy Measurements, and Overparameterization
Abstract
In this work, we study the performance of subgradient method (SubGM) on a natural nonconvex and nonsmooth formulation of lowrank matrix recovery with $\ell_1$loss, where the goal is to recover a lowrank matrix from a limited number of measurements, a subset of which may be grossly corrupted with noise. We study a scenario where the rank of the true solution is unknown and overestimated instead. The overestimation of the rank gives rise to an overparameterized model in which there are more degrees of freedom than needed. Such overparameterization may lead to overfitting, or adversely affect the performance of the algorithm. We prove that a simple SubGM with small initialization is agnostic to both overparameterization and noise in the measurements. In particular, we show that small initialization nullifies the effect of overparameterization on the performance of SubGM, leading to an exponential improvement in its convergence rate. Moreover, we provide the first unifying framework for analyzing the behavior of SubGM under both outlier and Gaussian noise models, showing that SubGM converges to the true solution, even under arbitrarily large and arbitrarily dense noise values, andperhaps surprisinglyeven if the globally optimal solutions do not correspond to the ground truth. At the core of our results is a robust variant of restricted isometry property, called SignRIP, which controls the deviation of the subdifferential of the $\ell_1$loss from that of an ideal, expected loss. As a byproduct of our results, we consider a subclass of robust lowrank matrix recovery with Gaussian measurements, and show that the number of required samples to guarantee the global convergence of SubGM is independent of the overparameterized rank.
 Publication:

arXiv eprints
 Pub Date:
 February 2022
 DOI:
 10.48550/arXiv.2202.08788
 arXiv:
 arXiv:2202.08788
 Bibcode:
 2022arXiv220208788M
 Keywords:

 Computer Science  Machine Learning;
 Mathematics  Optimization and Control;
 Statistics  Machine Learning