Scaling Nonparametric Bayesian Inference via SubsampleAnnealing
Abstract
We describe an adaptation of the simulated annealing algorithm to nonparametric clustering and related probabilistic models. This new algorithm learns nonparametric latent structure over a growing and constantly churning subsample of training data, where the portion of data subsampled can be interpreted as the inverse temperature beta(t) in an annealing schedule. Gibbs sampling at high temperature (i.e., with a very small subsample) can more quickly explore sketches of the final latent state by (a) making longer jumps around latent space (as in block Gibbs) and (b) lowering energy barriers (as in simulated annealing). We prove subsample annealing speeds up mixing time N^2 > N in a simple clustering model and exp(N) > N in another class of models, where N is data size. Empirically subsampleannealing outperforms naive Gibbs sampling in accuracyperwallclock time, and can scale to larger datasets and deeper hierarchical models. We demonstrate improved inference on millionrow subsamples of US Census data and network log data and a 307row hospital rating dataset, using a PitmanYor generalization of the Cross Categorization model.
 Publication:

arXiv eprints
 Pub Date:
 February 2014
 arXiv:
 arXiv:1402.5473
 Bibcode:
 2014arXiv1402.5473O
 Keywords:

 Statistics  Machine Learning;
 Statistics  Computation
 EPrint:
 To appear in AISTATS 2014