Critical Behavior and Universality Classes for an Algorithmic Phase Transition in Sparse Reconstruction
Abstract
Recovery of an N-dimensional, K-sparse solution x from an M-dimensional vector of measurements y for multivariate linear regression can be accomplished by minimizing a suitably penalized least-mean-square cost || y-{H} x||_2^2+λ V(x). Here H is a known matrix and V(x) is an algorithm-dependent sparsity-inducing penalty. For `random' H, in the limit λ → 0 and M,N,K→ ∞, keeping ρ =K/N and α =M/N fixed, exact recovery is possible for α past a critical value α _c = α (ρ ). Assuming x has iid entries, the critical curve exhibits some universality, in that its shape does not depend on the distribution of x. However, the algorithmic phase transition occurring at α =α _c and associated universality classes remain ill-understood from a statistical physics perspective, i.e. in terms of scaling exponents near the critical curve. In this article, we analyze the mean-field equations for two algorithms, Basis Pursuit (V(x)=||{x}||1 ) and Elastic Net (V({x})= ||{x}||1 + g/2 || x ||2^2) and show that they belong to different universality classes in the sense of scaling exponents, with mean squared error (MSE) of the recovered vector scaling as λ ^4/3 and λ respectively, for small λ on the critical line. In the presence of additive noise, we find that, when α >α _c, MSE is minimized at a non-zero value for λ , whereas at α =α _c, MSE always increases with λ.
- Publication:
-
Journal of Statistical Physics
- Pub Date:
- May 2019
- DOI:
- 10.1007/s10955-019-02292-6
- arXiv:
- arXiv:1509.08995
- Bibcode:
- 2019JSP...175..764R
- Keywords:
-
- High-dimensional data;
- Penalized regression;
- Cavity method;
- Phase transition;
- Random matrices;
- Compressed sensing;
- Computer Science - Information Theory;
- Condensed Matter - Statistical Mechanics
- E-Print:
- 18 pages, 8 figures, 3 tables