Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit
Abstract
In deep latent Gaussian models, the latent variable is generated by a timeinhomogeneous Markov chain, where at each time step we pass the current state through a parametric nonlinear map, such as a feedforward neural net, and add a small independent Gaussian perturbation. This work considers the diffusion limit of such models, where the number of layers tends to infinity, while the step size and the noise variance tend to zero. The limiting latent object is an Itô diffusion process that solves a stochastic differential equation (SDE) whose drift and diffusion coefficient are implemented by neural nets. We develop a variational inference framework for these \textit{neural SDEs} via stochastic automatic differentiation in Wiener space, where the variational approximations to the posterior are obtained by Girsanov (meanshift) transformation of the standard Wiener process and the computation of gradients is based on the theory of stochastic flows. This permits the use of blackbox SDE solvers and automatic differentiation for endtoend inference. Experimental results with synthetic data are provided.
 Publication:

arXiv eprints
 Pub Date:
 May 2019
 arXiv:
 arXiv:1905.09883
 Bibcode:
 2019arXiv190509883T
 Keywords:

 Computer Science  Machine Learning;
 Statistics  Machine Learning