Variational Autoencoder with Implicit Optimal Priors
Abstract
The variational autoencoder (VAE) is a powerful generative model that can estimate the probability of a data point by using latent variables. In the VAE, the posterior of the latent variable given the data point is regularized by the prior of the latent variable using Kullback Leibler (KL) divergence. Although the standard Gaussian distribution is usually used for the prior, this simple prior incurs overregularization. As a sophisticated prior, the aggregated posterior has been introduced, which is the expectation of the posterior over the data distribution. This prior is optimal for the VAE in terms of maximizing the training objective function. However, KL divergence with the aggregated posterior cannot be calculated in a closed form, which prevents us from using this optimal prior. With the proposed method, we introduce the density ratio trick to estimate this KL divergence without modeling the aggregated posterior explicitly. Since the density ratio trick does not work well in high dimensions, we rewrite this KL divergence that contains the highdimensional density ratio into the sum of the analytically calculable term and the lowdimensional density ratio term, to which the density ratio trick is applied. Experiments on various datasets show that the VAE with this implicit optimal prior achieves high density estimation performance.
 Publication:

arXiv eprints
 Pub Date:
 September 2018
 arXiv:
 arXiv:1809.05284
 Bibcode:
 2018arXiv180905284T
 Keywords:

 Statistics  Machine Learning;
 Computer Science  Machine Learning
 EPrint:
 9 pages, 9 figures, accepted at AAAI 2019. Code is available at https://github.com/takahashihiroshi/vae_iop