Kernels and Submodels of Deep Belief Networks
Abstract
We study the mixtures of factorizing probability distributions represented as visible marginal distributions in stochastic layered networks. We take the perspective of kernel transitions of distributions, which gives a unified picture of distributed representations arising from Deep Belief Networks (DBN) and other networks without lateral connections. We describe combinatorial and geometric properties of the set of kernels and products of kernels realizable by DBNs as the network parameters vary. We describe explicit classes of probability distributions, including exponential families, that can be learned by DBNs. We use these submodels to bound the maximal and the expected KullbackLeibler approximation errors of DBNs from above depending on the number of hidden layers and units that they contain.
 Publication:

arXiv eprints
 Pub Date:
 November 2012
 DOI:
 10.48550/arXiv.1211.0932
 arXiv:
 arXiv:1211.0932
 Bibcode:
 2012arXiv1211.0932M
 Keywords:

 Statistics  Machine Learning
 EPrint:
 13 pages, 4 figures, deep learning and unsupervised feature learning nips workshop 2012