The More, the Merrier: the Blessing of Dimensionality for Learning Large Gaussian Mixtures
Abstract
In this paper we show that very large mixtures of Gaussians are efficiently learnable in high dimension. More precisely, we prove that a mixture with known identical covariance matrices whose number of components is a polynomial of any fixed degree in the dimension n is polynomially learnable as long as a certain nondegeneracy condition on the means is satisfied. It turns out that this condition is generic in the sense of smoothed complexity, as soon as the dimensionality of the space is high enough. Moreover, we prove that no such condition can possibly exist in low dimension and the problem of learning the parameters is generically hard. In contrast, much of the existing work on Gaussian Mixtures relies on lowdimensional projections and thus hits an artificial barrier. Our main result on mixture recovery relies on a new "Poissonization"based technique, which transforms a mixture of Gaussians to a linear map of a product distribution. The problem of learning this map can be efficiently solved using some recent results on tensor decompositions and Independent Component Analysis (ICA), thus giving an algorithm for recovering the mixture. In addition, we combine our lowdimensional hardness results for Gaussian mixtures with Poissonization to show how to embed difficult instances of lowdimensional Gaussian mixtures into the ICA setting, thus establishing exponential informationtheoretic lower bounds for underdetermined ICA in low dimension. To the best of our knowledge, this is the first such result in the literature. In addition to contributing to the problem of Gaussian mixture learning, we believe that this work is among the first steps toward better understanding the rare phenomenon of the "blessing of dimensionality" in the computational aspects of statistical inference.
 Publication:

arXiv eprints
 Pub Date:
 November 2013
 arXiv:
 arXiv:1311.2891
 Bibcode:
 2013arXiv1311.2891A
 Keywords:

 Computer Science  Machine Learning;
 Computer Science  Data Structures and Algorithms;
 Statistics  Machine Learning
 EPrint:
 29 pages