Abstract
Particle filters contain the promise of fully nonlinear data assimilation. They have been applied in numerous science areas, but their application to the geosciences has been limited due to their inefficiency in high-dimensional systems in standard settings. However, huge progress has been made, and this limitation is disappearing fast due to recent developments in proposal densities, the use of ideas from (optimal) transportation, the use of localisation and intelligent adaptive resampling strategies. Furthermore, powerful hybrids between particle filters and ensemble Kalman filters and variational methods have been developed. We present a state of the art discussion of present efforts of developing particle filters for highly nonlinear geoscience state-estimation problems with an emphasis on atmospheric and oceanic applications, including many new ideas, derivations, and unifications, highlighting hidden connections, and generating a valuable tool and guide for the community. Initial experiments show that particle filters can be competitive with present-day methods for numerical weather prediction suggesting that they will become mainstream soon.
Introduction: The Challenge of Nonlinear Data Assimilation
In the geosciences, data assimilation is the critical process of merging numerical models (e.g., weather forecasts) with real-world observations. Current operational standards rely heavily on Variational methods (4DVar) or Ensemble Kalman Filters (EnKF). While effective, these methods typically assume linear system dynamics and Gaussian (bell-curve) uncertainty distributions.
However, Earth systems are inherently messy. At convective scales—such as individual clouds or ocean eddies—physics are highly nonlinear and non-Gaussian. Particle Filters (PFs) are theoretically ideal for these complexities. The primary obstacle has been "weight collapse" in high-dimensional systems (often exceeding $10^9$ variables). In such cases, a single particle often captures all the weight, rendering the ensemble useless unless an unfeasibly large number of particles is used.
Methodological Innovations
To overcome weight collapse, researchers have developed four primary strategies:
- Proposal Densities: Instead of random wandering, "proposals" steer particles toward observations, ensuring they occupy high-probability regions. Techniques like the Equivalent-Weights Particle Filter (EWPF) and the Implicit Particle Filter are leading examples.
- Transportation Particle Filters: Leveraging optimal transport theory, filters like the ETPF move particles deterministically from prior to posterior states, minimizing movement distance rather than relying solely on reweighting.
- Localisation: Borrowed from EnKF success, localisation applies filters to small spatial patches. By assuming an observation in one region (e.g., New York) does not immediately affect a distant one (e.g., London), the "effective" dimensionality is drastically reduced.
- Hybrid Systems: These combine EnKF robustness with PF flexibility. An EnKF might handle the bulk of the distribution, while the PF captures non-Gaussian "tails."
Key Insights and Experimental Results
Research indicates that even "optimal" proposal densities collapse globally in high dimensions; thus, localisation is mandatory for weather-scale models. While new "equal-weight" filters (like IEWPF) prevent collapse by design, they introduce a manageable bias—a necessary tradeoff for high-dimensional stability.
A significant challenge remains in maintaining physical balance. Local resampling can introduce physical discontinuities (e.g., spurious pressure jumps). Advanced "gluing" or smoothing techniques are essential for global consistency. Furthermore, "Particle Flow" filters are creating a new synergy between data assimilation and machine learning.
In terms of performance, the Localised Adaptive Particle Filter (LAPF) has been tested in the ICON global weather model with 40 particles at 40km resolution. Results show PFs are currently only 10-15% behind state-of-the-art operational systems like LETKF-EnVAR. In scenarios with highly nonlinear observations or multimodal distributions, PFs already significantly outperform traditional Kalman filters.
Future Directions
Despite their promise, PFs face challenges regarding mathematical consistency—many high-dimensional "tricks" prevent convergence to the true posterior with infinite particles. Additionally, computational costs remain higher than standard EnKFs. Future work will likely focus on refining "Particle Flow" methods and sophisticated hybrids to bring particle filters into the mainstream of global numerical weather prediction.
Summary updated