A Particle Flow data-assimilation method for high-dimensional systems

Many geophysical systems are highly nonlinear, asking for nonlinear data-assimilation methods. Particle filters, after a rough start, are slowly becoming mainstream, evidenced by the recent implementation by DWD of a localized particle filter for global numerical weather prediction. However, as all data-assimilation methods at this scale of operation, localized particle filters need ad-hoc tricks to avoid weight collapse, such as setting a minimal weight, or projecting observations onto the ensemble space. Particle Flows are particle filters in which the samples from the prior are transformed via a flow in state space to samples from the posterior, without the need for introducing weights. Hence Particle Flows are equal-weight particle filters by construction. However, using them with a small number of particles remains challenging. By exploring ideas from machine learning, such as kernel embedding, kernel gradient estimation and stochastic gradient descent we develop a new method that is extremely promising for high-dimensional applications with a small number of particles. We will report on applying this technique to a high-dimensional atmospheric example, discussing details of the implementation, results, and remaining issues. These are exciting times for nonlinear data assimilation and is meant to be a strong contribution to that effort.