Statistical mechanics of sparse generalization and graphical model selection
Abstract
One of the crucial tasks in many inference problems is the extraction of an underlying sparse graphical model from a given number of high-dimensional measurements. In machine learning, this is frequently achieved using, as a penalty term, the Lp norm of the model parameters, with p<=1 for efficient dilution. Here we propose a statistical mechanics analysis of the problem in the setting of perceptron memorization and generalization. Using a replica approach, we are able to evaluate the relative performance of naive dilution (obtained by learning without dilution, following by applying a threshold to the model parameters), L1 dilution (which is frequently used in convex optimization) and L0 dilution (which is optimal but computationally hard to implement). Whereas both Lp diluted approaches clearly outperform the naive approach, we find a small region where L0 works almost perfectly and strongly outperforms the simpler to implement L1 dilution.
- Publication:
-
Journal of Statistical Mechanics: Theory and Experiment
- Pub Date:
- October 2009
- DOI:
- 10.1088/1742-5468/2009/10/P10009
- arXiv:
- arXiv:0907.3241
- Bibcode:
- 2009JSMTE..10..009L
- Keywords:
-
- Condensed Matter - Disordered Systems and Neural Networks;
- Condensed Matter - Statistical Mechanics;
- Physics - Data Analysis;
- Statistics and Probability
- E-Print:
- 18 pages, 9 eps figures