Towards Generalizable Neural Simulators: Addressing Distribution Shifts Induced by Environmental and Temporal Variations

Jiaqi Liu, Jiaxu Cui, Shiang Sun, Yizhu Zhao, Bo YangIJCAI 2025 · DOI

TL;DR

This paper introduces CoPoNDP, a novel neural simulator designed to tackle the challenge of modeling complex dynamical systems that experience both environmental and temporal distribution shifts. By employing a context-informed, stochastic process approach, CoPoNDP demonstrates superior generalization capabilities across diverse scenarios, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Simulating complex dynamical systems is crucial across various scientific and technological fields. Traditional numerical simulators often struggle with real-world complexities, being too idealized or computationally intensive. Neural simulators have emerged as a powerful alternative, leveraging deep learning to achieve better accuracy and efficiency. However, a significant challenge remains: these simulators often assume data is independently and identically distributed (i.i.d.), an assumption that rarely holds in practice. Real-world systems are frequently influenced by external factors that cause "distribution shifts." These shifts can arise from:

  • Environmental Variations: Changes in external conditions (e.g., different cities for power demand, varying climate) lead to different data distributions.
  • Temporal Variations: System behavior changes over time due to evolving conditions (e.g., daily fluctuations in electricity usage).

Existing neural simulators have made progress in addressing either environmental or temporal shifts individually. However, many real-world applications exhibit both types of shifts concurrently, making it difficult for current models to generalize effectively. This paper aims to bridge this gap by proposing a unified framework that can handle these dual distribution shifts simultaneously.

Methods

The core of the proposed solution is the Context-informed Polymorphic Neural ODE Processes (CoPoNDP). This framework is built upon a "sandwich" architecture of encoders, processors, and a decoder.

  • Encoders: These components take contextual information (observed system states) as input and generate probability distributions for:
  • Initial latent states: The starting point of the system's hidden representation.
  • Initial time-varying combination coefficients: Parameters that will modulate how different base dynamics are combined over time.
  • Random control vector: This vector encodes environmental context and influences how the system evolves.
  • Processors: This is where the dynamic evolution happens.
  • Latent State Evolution: The latent state $l(t)$ is evolved using a weighted combination of $K$ base neural Ordinary Differential Equations (ODEs). This allows for flexible modeling of complex dynamics.
  • Time-Varying Coefficients Evolution: The coefficients $a(t)$ that weight the base ODEs are themselves evolved using a neural ODE. Crucially, this evolution is influenced by the control vector $d$, allowing the model to adapt to environmental changes. This dynamic modulation of base ODEs is key to handling temporal shifts.
  • Decoder: This component maps the evolved latent state back to the observable system states, providing a predictive distribution.

The model is trained end-to-end using variational inference to maximize a lower bound, enabling it to learn from sparse and noisy data and generalize to unseen environments.

Datasets

The paper evaluates CoPoNDP on four representative complex dynamical systems:

  • Pendulum with external control forces (PE)
  • Ecosystem with time-varying birth and death rates (LV)
  • Controlled vibration of a string (ST)
  • Real-world datasets: Power system demand forecasting (PW), exchange prediction (EX), air quality (AQ), and weather forecasting (WT).

The experiments involve creating various testing scenarios, including:

  • In-Distribution Test: Stable conditions.
  • Environmental Adaptation Test: Changes in environment, but not time.
  • Temporal Adaptation Test: Changes in time, but not environment.
  • Joint-Distribution Test: Simultaneous environmental and temporal shifts.

Key Insights and Ideas

  • Unified Framework for Dual Shifts: The most significant contribution is the development of a single model capable of handling both environmental and temporal distribution shifts simultaneously, a problem that has largely been addressed in isolation.
  • Stochastic Process Perspective: Framing the simulator as a neural-network-parameterized stochastic process provides theoretical grounding (exchangeability and consistency) and inherent robustness to uncertainty.
  • Contextual Modulation of Dynamics: The core innovation lies in using context (observed states) to dynamically modulate the evolution of system dynamics through time-varying coefficients that influence a combination of base ODEs. This allows for adaptive behavior.
  • Robustness to Sparsity and Noise: The probabilistic nature and context-driven approach make CoPoNDP resilient to incomplete or noisy observations, a common issue in real-world data.
  • Flexibility in Dynamics Modeling: The use of multiple base ODEs (branches) allows for capturing diverse dynamics, with an optimal number of branches found to balance expressiveness and computational cost.

Key Results

  • Superior Performance on Joint Distribution Shifts: CoPoNDP significantly outperforms state-of-the-art baselines, including models specialized for environmental or temporal adaptation, on the challenging Joint-Distribution Test. For instance, it reduces error by an order of magnitude for the PE and LV systems.
  • Competitive Performance on Single Shifts: CoPoNDP also achieves competitive or superior results on tests focusing solely on environmental or temporal adaptation.
  • Robustness to Sparsity and Noise: Even with up to 90% data sparsity and 10% noise, CoPoNDP shows limited performance degradation, outperforming baselines like LEADS under similar conditions.
  • Adaptation to Streaming Data: The model can adapt to new data in real-time without retraining, progressively refining predictions and reducing uncertainty as more context becomes available.

Limitations and Future Work

  • Computational Overhead: While CoPoNDP offers significant advantages, the paper notes that scaling to real-world applications may require further advances in automated architecture optimization and physics-aware efficiency improvements.
  • Number of Branches: The study found an optimal number of base ODE branches (around 5-6) for balancing performance and complexity. Further research could explore more sophisticated methods for automatically determining or adapting this number.
  • Future Directions: The authors suggest exploring automated architecture optimization and physics-aware efficiency improvements for scaling to larger, more complex real-world systems. The work also lays a foundation for developing more generalizable neural simulators that can adapt to evolving dynamics while maintaining computational tractability.

Summary updated

BibTeX
@inproceedings{liu2025towards,
  author = {Jiaqi Liu and Jiaxu Cui and Shiang Sun and Yizhu Zhao and Bo Yang},
  title = {{Towards Generalizable Neural Simulators: Addressing Distribution Shifts Induced by Environmental and Temporal Variations}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/841},
}
Figure