Neural Conditional Gaussian Model: Efficient Data Assimilation with Sequential Uncertainty Quantification for Nonlinear Dynamics

Chuanqi Chen, Nan ChenarXiv 2026 · arXiv

TL;DR

Neural CGM is a data-driven framework that enables fast, analytical data assimilation for complex nonlinear systems. By learning a latent space with a "conditional Gaussian" structure, it provides accurate state forecasts and real-time uncertainty quantification with up to a 400x speedup over traditional particle-based methods.

What Problem Does This Paper Address?

In fields like meteorology, fluid dynamics, and aerospace, we often deal with "partially observed" systems—we can see some variables (like surface temperature) but not others (like deep ocean currents). Data Assimilation (DA) is the process of combining these sparse observations with a physical model to estimate the full state of the system and, crucially, the uncertainty of that estimate.

Traditional nonlinear filtering methods face a "pick your poison" scenario:

  1. Classical Filters (EKF/UKF): Use linear approximations that fail in highly chaotic systems.
  2. Ensemble/Particle Filters (EnKF/PF): Are more accurate but computationally grueling, requiring hundreds or thousands of model simulations to represent uncertainty.
  3. Model Dependency: Most methods require an explicit, accurate physical model (governing equations), which may be unknown or too expensive to run repeatedly.

The authors aim to create a model that is entirely data-driven, captures complex nonlinearities, and performs the mathematical heavy lifting of uncertainty propagation analytically rather than through brute-force simulation.

Methods

The core of the paper is the Neural Conditional Gaussian Model (Neural CGM). Its architecture consists of three main components:

  1. Encoder-Decoder Framework: A nonlinear encoder maps unobserved physical states into a latent space, while a decoder maps them back.
  2. Conditionally Gaussian Latent Dynamics: This is the "secret sauce." The dynamics in the latent space are constrained to be linear with respect to the unobserved variables, conditioned on the observations. This specific structure allows the model to use a Conditional Gaussian (CG) filter—a set of analytical equations that update the mean and covariance of the state in closed form, similar to a Kalman filter but for nonlinear systems.
  3. Distribution-Aware Decoder: Unlike previous latent-space models that only decode the "best guess" (mean), Neural CGM uses a specialized decoder that takes both the latent mean and the latent covariance as inputs. It uses the Jacobian of the decoder and learned nonlinear corrections to map latent uncertainty directly into physical-space uncertainty.
Figure 2.1: Overview of Neural CGM for point forecasting and distribution-aware data assimilation. The forecasting branch maps predictive latent states to physical states. The DA branch updates the latent posterior mean and co- variance analytically and maps both quantities to physical-space posterior statistics, preserving sequential uncertainty information.

The model is trained using a multi-objective loss function that balances reconstruction accuracy, forecasting precision, and a Data Assimilation Negative Log-Likelihood (NLL). This ensures the model is optimized not just to predict the state, but to get the "error bars" right.

Datasets

The authors evaluated Neural CGM on three classic benchmarks for chaotic and nonlinear dynamics:

  • Lorenz 63: A 3-dimensional chaotic system used to test basic DA performance.
  • Lorenz 96: A 40-dimensional system that mimics atmospheric waves, representing a higher-dimensional challenge.
  • Kuramoto-Sivashinsky (KS) Equation: A 128-dimensional spatiotemporal partial differential equation (PDE) that represents complex, turbulent-like structures.

Key Insights and Ideas

  • Analytical Tractability vs. Expressivity: You don't need a fully linear model to get fast, analytical DA. By enforcing "conditional linearity" in a learned latent space, you can keep the flexibility of deep learning while regaining the speed of classical filters.
  • Uncertainty as a First-Class Citizen: By including the DA NLL in the training objective, the neural network is forced to learn the underlying noise and error structures of the system. The uncertainty isn't a post-processing step; it's baked into the model's DNA.
  • Decoupling Forecast and Inference: The model uses a "Forecast Decoder" for deterministic predictions and a "Distribution-Aware Decoder" for probabilistic inference. This separation allows the model to excel at both tasks without one compromising the other.

Key Results

Neural CGM demonstrates a massive leap in computational efficiency without sacrificing accuracy:

  • Massive Speedups: On the complex KS equation, Neural CGM was approximately 400 times faster than a Particle Filter (PF) while achieving lower error (RMSE).
  • Superior Uncertainty Quantification: In higher-dimensional tests (Lorenz 96 and KS), Neural CGM provided more concentrated and accurate posterior distributions (lower NLL) compared to other machine learning frameworks like CGKN.
  • Competitive Accuracy: Despite being data-driven, it performed comparably to (and sometimes better than) classical methods like EnKF that had the advantage of knowing the true governing equations.

Limitations and Future Work

  • Training Sensitivity: Training involves balancing three different loss terms, which requires careful hyperparameter tuning and can sometimes lead to unstable solutions.
  • Data Requirements: The current "offline" training phase requires access to the full state (both observed and unobserved variables). In many real-world applications, we may never have the "ground truth" for unobserved variables, suggesting a need for semi-supervised versions of this framework.
  • Covariance Simplification: To save on costs, the current implementation uses diagonal physical-space covariances. Future iterations could incorporate structured or full covariances to better capture correlations between different physical variables.

Summary updated

Abstract

Data assimilation (DA) in partially observed nonlinear systems requires repeated estimation of both hidden states and their uncertainty. Conventional nonlinear filtering can be computationally expensive and often relies on an explicit dynamical model. In this paper, we develop the Neural Conditional Gaussian Model (Neural CGM), a data-driven dynamical model that maps unobserved physical states into a learned latent space, where the dynamics of the latent states follow a conditional Gaussian structure. This structure allows closed-form DA updates in the latent space, while nonlinear encoder-decoder mappings retain the flexibility needed to represent complex dynamics in physical space. A key feature of Neural CGM is its treatment of uncertainty in DA. The latent posterior mean and covariance are mapped to the physical variables through a Gaussian decoder that combines local moment propagation with learned nonlinear corrections. The resulting physical-space posterior statistics depend on the sequentially updated latent distribution rather than on a separate post-processing estimate. The same statistics enter a Gaussian negative log-likelihood during training, so both the posterior mean and uncertainty affect the learned model. Because Neural CGM learns the dynamical model itself, it can also be used for state forecasting and simulation without explicit knowledge of the governing equations. Numerical experiments on benchmark nonlinear dynamical systems show accurate state estimation and forecasting while substantially reducing online assimilation cost relative to conventional nonlinear DA methods. Neural CGM is therefore suited to repeated state estimation, uncertainty quantification, and prediction from partial observations.
BibTeX
@misc{chen2026neural,
  author = {Chuanqi Chen and Nan Chen},
  title = {{Neural Conditional Gaussian Model: Efficient Data Assimilation with Sequential Uncertainty Quantification for Nonlinear Dynamics}},
  year = {2026},
  eprint = {2611.26463},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26463},
}
Figure