A data-driven statistical-stochastic surrogate modeling strategy for complex nonlinear non-stationary dynamics

Di Qi, John HarlimJournal of Computational Physics 2023 · DOI

TL;DR

This paper introduces a novel data-driven approach to create surrogate models for complex, non-stationary dynamical systems. By combining statistical and stochastic modeling with machine learning (specifically LSTMs), they can accurately predict the mean and variance of system behavior, even in challenging, inhomogeneous regimes, overcoming limitations of previous methods.

What Problem Does This Paper Address?

Predicting the behavior of complex dynamical systems, especially those with non-linear, non-stationary dynamics, is a significant challenge in many scientific fields. Accurately quantifying uncertainty in these systems is crucial for applications like weather forecasting and climate modeling. Traditional methods like Monte Carlo simulations are computationally expensive, especially for high-dimensional systems, as the required ensemble size grows exponentially with dimensionality. Existing statistical surrogate models struggle with non-homogeneous statistical regimes and maintaining positive-definite covariance matrices. This paper aims to develop a more efficient and accurate method to capture the low-order statistics (mean and variance) of these complex systems, even when they exhibit inhomogeneous behavior.

Methods

The authors propose a "statistical-stochastic surrogate modeling strategy." This involves:

  • Statistical-Stochastic Decomposition: Decomposing the system's state into a mean component and stochastic fluctuations.
  • Reduced-Order Modeling: Projecting the system's dynamics onto a lower-dimensional subspace defined by the most energetic modes.
  • Machine Learning for Closure: Employing Long-Short-Term-Memory (LSTM) neural networks to learn the "unresolved fluxes" – the feedback from higher-order statistical moments that are not explicitly modeled. This is a key innovation, extending previous work to handle non-homogeneous statistics.
  • Effective Damping and Noise Parameterization: Introducing a method to parameterize effective damping and noise terms to ensure the stability of the learned model, which is crucial for chaotic systems.
  • Information Metric-Based Loss Function: Using an information-theoretic loss function (specifically, a KL divergence-based metric) to train the model. This focuses on fitting the response statistics (mean and variance) rather than individual trajectories, which is more robust and less prone to overfitting.

Datasets

The authors use the Lorenz-96 (L-96) model as a representative test case. They generate synthetic datasets by simulating the L-96 system under various homogeneous and inhomogeneous forcing and damping conditions. These simulations provide the "ground truth" statistics (mean and variance) used for training and evaluating their surrogate model. No external real-world datasets are used.

Key Insights and Ideas

  • Focusing on Statistics, Not Trajectories: The most surprising and impactful insight is that fitting the model to statistical responses (mean and variance) rather than individual trajectories leads to much more accurate predictions, especially for new, unseen initial conditions. This directly addresses the overfitting problem common in trajectory-based learning.
  • Handling Inhomogeneity is Key: The proposed method's ability to handle non-homogeneous statistics, which are common in real-world systems (like weather patterns), is a significant advancement over previous homogeneous-only approaches.
  • Stability Through Parameterization: The explicit parameterization of effective damping and noise is crucial for learning stable models of inherently unstable chaotic systems. Without this, even accurate statistical predictions can quickly diverge.
  • Information Metric for Robustness: Using an information-theoretic loss function provides a more principled way to measure the difference between statistical distributions, leading to better generalization.

Key Results

  • The proposed statistical-stochastic surrogate model accurately predicts the mean and variance statistics of the Lorenz-96 system for both homogeneous and inhomogeneous regimes.
  • The model demonstrates good prediction skill even for initial conditions and forcing perturbations not seen during training.
  • The approach is shown to be more robust than direct trajectory fitting, which often leads to overfitting and poor generalization.
  • The method effectively captures the dynamics of complex systems with strong instability and non-Gaussian statistics.

Limitations and Future Work

  • Non-Diagonal Covariance: While the model performs well, the authors acknowledge that the parameterization of the unresolved feedback (QF) ignores non-diagonal components of the covariance matrix. This could potentially lead to instability for longer prediction times, especially if conditional linear instability is present.
  • Long-Term Stability: Guaranteeing stability for arbitrarily long prediction times remains an open issue, particularly concerning the impact of learning errors and temporal discretization.
  • Generalization to Other Systems: While the L-96 model is a good test case, further validation on other complex, non-stationary systems would be beneficial.
  • Improved Parameterization: Developing improved parameterizations for QF that account for non-diagonal components and guarantee long-term stability is a promising direction for future work. The authors also suggest exploring the validity of linear response theory for chaotic systems to better understand the behavior of perturbed dynamical systems.

Summary updated

BibTeX
@article{qi2023data,
  author = {Di Qi and John Harlim},
  title = {{A data-driven statistical-stochastic surrogate modeling strategy for complex nonlinear non-stationary dynamics}},
  journal = {Journal of Computational Physics},
  year = {2023},
  doi = {10.1016/j.jcp.2023.112085},
}
Figure