Particle filters for high-dimensional geoscience applications: a review

Peter Jan van Leeuwen, Hans R. Künsch, Lars Nerger, Roland Potthast, Sebastian Reich

Abstract

Particle filters contain the promise of fully nonlinear data assimilation. They have been applied in numerous science areas, but their application to the geosciences has been limited due to their inefficiency in high-dimensional systems in standard settings. However, huge progress has been made, and this limitation is disappearing fast due to recent developments in proposal densities, the use of ideas from (optimal) transportation, the use of localisation and intelligent adaptive resampling strategies. Furthermore, powerful hybrids between particle filters and ensemble Kalman filters and variational methods have been developed. This review presents a state-of-the-art discussion of present efforts in developing particle filters for highly nonlinear geoscience state-estimation problems, highlighting hidden connections and generating a valuable tool for the community.

Introduction: The Data Assimilation Challenge

Data assimilation (DA) is the critical process of combining mathematical model forecasts with real-world observations. In geosciences—such as meteorology and oceanography—this task is exceptionally difficult because systems are massive (often involving billions of variables) and highly nonlinear.

Currently, the field relies heavily on Variational methods (4DVar) or Ensemble Kalman Filters (EnKF). While effective, these methods typically assume that errors follow a Gaussian (bell curve) distribution and that relationships between variables are linear. When these assumptions break down—such as during intense localized storms or at very high model resolutions—standard methods struggle to provide accurate estimates.

The Curse of Dimensionality and Weight Collapse

Particle filters (PFs) offer a "fully nonlinear" alternative that does not require Gaussian assumptions. However, in high-dimensional spaces, standard PFs suffer from a phenomenon known as weight collapse.

Illustration of weight collapse: in high-dimensional spaces, a single particle often ends up with all the mathematical weight, rendering the ensemble ineffective.

Without a massive, often computationally impossible number of "particles" (model realizations), the mathematical weight concentrates on a single realization. This failure means the filter loses its ability to represent the true probability distribution of the system state.

Modern Strategies to Overcome Collapse

Recent breakthroughs have categorized progress into four main strategies to make PFs viable for massive systems:

  1. Proposal Densities: Instead of letting particles evolve blindly, "intelligent" proposal densities steer particles toward observations. The Equivalent-Weights Particle Filter (EWPF), for instance, moves particles to specific positions to ensure they maintain similar weights.
  2. Transportation Particle Filters: Based on optimal transport theory, these methods move particles from the "prior" (before observations) to the "posterior" (after observations) using deterministic maps or "particle flows" rather than simple reweighting.
  3. Localization: This involves calculating weights based only on nearby observations. While common in EnKFs, it is challenging in PFs because "glueing" locally resampled particles can create unphysical discontinuities in the model state.
  4. Hybrid Methods: These combine the robustness of EnKFs with the nonlinear flexibility of PFs. An EnKF might handle the broad state estimation, while a PF step manages highly non-Gaussian features.
Visualization of particle trajectories and selection from Time n-1 to Time n.

Key Insights from Recent Research

The review highlights several "aha moments" for the community:

  • Manageable Ensemble Sizes: The "exponential growth" of particles required is a property of standard filters, not an inherent limit of the PF concept itself. Intelligent steering makes small ensembles feasible.
  • The "Glueing" Problem: Localization isn't just about local weighting; it's about ensuring the resulting global state remains physically balanced (e.g., maintaining atmospheric pressure balances).
  • Acceptable Bias: Many successful high-dimensional PFs are technically "biased," meaning they don't converge to the exact mathematical posterior. However, in practice, this bias is often much smaller than the noise (variance) found in "unbiased" filters with small ensembles.

Operational Results and Future Outlook

The transition from theory to practice is already underway. The Localized Adaptive Particle Filter (LAPF) has been successfully tested in the global ICON model (used by the German Weather Service) with 40 particles at 40km resolution.

While these new particle filters currently trail state-of-the-art Ensemble-Variational (EnVar) systems by about 10–15% in forecast scores, they are rapidly closing the gap. They show particular promise in systems with highly non-Gaussian features, such as clouds and precipitation.

The field remains young, and challenges like tuning complexity and maintaining physical balances persist. However, the emergence of "Data Assimilation as a Flow"—viewing the process as a continuous movement of particles—suggests that particle filters will soon become a mainstream tool in numerical weather prediction.

Summary updated

Abstract

Particle filters contain the promise of fully nonlinear data assimilation. They have been applied in numerous science areas, but their application to the geosciences has been limited due to their inefficiency in high-dimensional systems in standard settings. However, huge progress has been made, and this limitation is disappearing fast due to recent developments in proposal densities, the use of ideas from (optimal) transportation, the use of localisation and intelligent adaptive resampling strategies. Furthermore, powerful hybrids between particle filters and ensemble Kalman filters and variational methods have been developed. We present a state of the art discussion of present efforts of developing particle filters for highly nonlinear geoscience state-estimation problems with an emphasis on atmospheric and oceanic applications, including many new ideas, derivations, and unifications, highlighting hidden connections, and generating a valuable tool and guide for the community. Initial experiments show that particle filters can be competitive with present-day methods for numerical weather prediction suggesting that they will become mainstream soon.
BibTeX
@inproceedings{leeuwen2019particle,
  author = {Peter Jan van Leeuwen and Hans R. Künsch and Lars Nerger and Roland Potthast and Sebastian Reich},
  title = {{Particle filters for high-dimensional geoscience applications: a review}},
  booktitle = {Quarterly},
  year = {2019},
  doi = {10.1002/qj.3551},
}
Figure