Stochastic Neural Simulator for Generalizing Dynamical Systems across Environments

Jiaqi Liu, Jiaxu Cui, Jiayi Yang, Bo YangIJCAI 2024

Introduction

Modeling complex dynamical systems—ranging from fluid dynamics to biological oscillations—is fundamental to scientific discovery. While deep learning has enabled the creation of "neural simulators" capable of approximating these systems, most existing models rely on the assumption of abundant, independent, and identically distributed (i.i.d.) data from a single environment.

In practice, however, physical environments are rarely static. Factors like varying temperature, pressure, or gravity alter system behavior. Furthermore, data acquisition is often limited by sensor failures or high costs, resulting in "sparse" datasets. Traditional cross-environmental models, such as LEADS and CoDA, are deterministic; they attempt to identify a single set of governing equations. In sparse data regimes, this approach is prone to overfitting noise or failing to generalize, as multiple equations might equally fit the limited observations.

The CoNDP Framework

To address these challenges, the authors propose Context-Informed Neural ODE Processes (CoNDP). Unlike its predecessors, CoNDP treats the learning of dynamical systems as a stochastic process, allowing it to model uncertainty and generalize more effectively across diverse environments.

Comparison of CoNDP with deterministic baselines LEADS and CoDA. CoNDP introduces a stochastic environment representation via a random control vector $u^e$.

As shown in the comparison above, while LEADS and CoDA focus on deterministic parameter tuning per environment, CoNDP utilizes a context-informed encoder and a latent distribution to capture environmental variability.

System Architecture

The CoNDP architecture is built upon four primary components designed to handle uncertainty and extract environmental features implicitly:

  1. Initial Encoder: Rather than predicting a single starting state, this module generates a Gaussian distribution for the initial latent state ($z_{t0}$). This allows the model to quantify uncertainty regarding the system's origin.
  2. Context-Informed Encoder: This component acts as the system's perceptual unit. It employs a temporal self-attention mechanism on historical trajectories (the "context") to identify latent environmental features. A hyper-network dynamically generates environment-specific attention weights, enabling the model to adapt its focus based on the observed behavior.
  3. Stochastic Environment Representation: The environment is represented by a random control vector ($u^e$). By sampling from this distribution, the simulator can explore a range of governing equations consistent with sparse observations, rather than being locked into a single potentially incorrect fit.
  4. Neural ODE Solver: The latent state evolves over time via an ODE solver. The governing function $f$ is decomposed into shared components (universal physical laws) and environment-specific components (local variations) driven by the control vector.
The full architectural pipeline of CoNDP, illustrating the flow from context and initial state through the hyper-network and ODE solver to the final predictions.

Experimental Evaluation

CoNDP was rigorously tested against five diverse dynamical systems:

  • Lotka-Volterra (LV): Biological predator-prey dynamics.
  • Glycolitic-Oscillator (GO): Metabolic oscillations in yeast.
  • Gray-Scott (GS): Chemical reaction-diffusion patterns.
  • Navier-Stokes (NS): Fundamental fluid dynamics.
  • Heat Diffusion (HEAT): Thermal energy propagation.

The evaluation covered both transductive settings (predicting future states in known environments) and inductive settings (generalizing to entirely new environments with different physical constants).

Key Insights and Results

The research yielded several critical findings:

  • Accuracy and Robustness: CoNDP consistently outperformed state-of-the-art models, reducing Mean Square Error (MSE) by 3.15% to 8.88% compared to CoDA. Its advantage was most pronounced in high-sparsity scenarios (as low as 4% data density), where deterministic models typically diverged.
  • Latent Interpretability: T-SNE visualizations revealed that the learned latent vectors ($u^e$) naturally clustered according to physical parameters like reaction speeds or birth rates. This suggests the model successfully captures underlying physical properties without explicit supervision.
  • Few-Shot Adaptation: Due to the low-dimensional nature of environment-specific parameters ($\xi^e$) and the efficiency of the hyper-network, CoNDP can adapt to novel environments using only a minimal number of observations.
  • Discovery of Governing Laws: When integrated with symbolic regression tools like SINDy, CoNDP proved more effective at recovering the actual mathematical equations of the underlying systems than previous neural simulators.

Conclusion and Future Directions

CoNDP represents a significant step toward robust neural simulation in real-world, data-scarce conditions. By embracing stochasticity, it provides a more reliable framework for cross-environmental generalization.

Future work aims to extend this framework to "counterfactual" reasoning—predicting system changes when variables are altered mid-simulation—and integrating these simulators into real-time control systems for AI agents. Additionally, optimizing the latent ODE processes for massive, high-dimensional systems like global climate models remains a key challenge for scaling this technology.

Summary updated

BibTeX
@inproceedings{liu2024stochastic,
  author = {Jiaqi Liu and Jiaxu Cui and Jiayi Yang and Bo Yang},
  title = {{Stochastic Neural Simulator for Generalizing Dynamical Systems across Environments}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure