True Zero-Shot Inference of Dynamical Systems Preserving Long-Term Statistics

Christoph Jürgen Hemmer, Daniel Durstewitz

Abstract

Recent advancements in time-series (TS) foundation models have significantly improved short-term forecasting capabilities. However, these models often struggle with Dynamical Systems Reconstruction (DSR)—the ability to capture the underlying generative laws, long-term invariants, and attractor geometries of a system. We introduce DynaMix, a foundation model designed for true zero-shot reconstruction of complex dynamical systems. By employing a lightweight mixture-of-experts (MoE) architecture based on Almost-Linear Recurrent Neural Networks (ALRNN), DynaMix faithfully captures the "climate" of unseen systems. Despite having only ~10,000 parameters—making it 1,000x smaller than state-of-the-art models like Chronos—DynaMix demonstrates superior performance in preserving long-term statistics and geometric stability across both synthetic and real-world datasets.

1. Introduction

In time-series analysis, a critical divide exists between short-term forecasting and Dynamical Systems Reconstruction (DSR). While forecasting focuses on predicting the next few steps in a sequence, DSR aims to build a generative model that captures the "soul" of a system—its attractor shapes, power spectra, and long-term invariants.

Current TS foundation models, such as Amazon’s Chronos or Google’s TimesFM, excel at short-term "look-ahead" tasks but frequently fail at DSR. These models often suffer from "context parroting" (repeating input patterns without understanding the underlying rules) or decaying into unrealistic fixed points during long-term generation. Furthermore, most existing DSR methods require specialized training for every new system observed. DynaMix addresses these challenges by providing a zero-shot framework capable of inferring universal principles of dynamics from short data snippets.

2. The DynaMix Architecture

DynaMix utilizes a multivariate architecture centered on the Almost-Linear Recurrent Neural Network (ALRNN). Its design is governed by four key technical components:

  • Mixture-of-Experts (MoE): Rather than relying on a single monolithic network, DynaMix employs 10 specialized ALRNN experts. A gating network, comprising a CNN and a state-attention mechanism, analyzes the context signal to determine the optimal combination of experts for the observed dynamics.
  • Sparse Teacher Forcing (STF): Training on chaotic systems is notoriously difficult due to the exponential explosion of small errors. STF serves as a critical training intervention, recalibrating the model's trajectory at optimal intervals. This allows the model to explore future states during training without gradient instability.
  • Multivariate Coupling: Unlike many TS models that treat dimensions independently, DynaMix is inherently multivariate. It captures the essential dependencies between variables that define a system's state space.
  • State-Space Embedding: To process real-world 1D data, the model utilizes temporal delay embedding or positional encoding to "lift" signals into a higher-dimensional space, clarifying the underlying dynamics.

3. Experimental Setup

3.1 Training

Remarkably, DynaMix was trained on a small corpus of only 34 synthetic 3D dynamical systems, including both chaotic and cyclic regimes. This small training set highlights the model's ability to learn fundamental physical and mathematical priors rather than relying on massive data scaling.

3.2 Evaluation

The model was evaluated across a diverse battery of unseen systems:

  • Synthetic Systems: 54 unseen 3D systems, 2D systems (e.g., Selkov oscillator), and higher-dimensional systems (e.g., 6D Lorenz-96).
  • Real-World Data: Hourly traffic patterns, solar temperatures, air pressure, cloud service requests, and human physiological signals (fMRI and EEG).

4. Key Insights and Results

4.1 Efficiency and Scale

DynaMix achieves state-of-the-art results with approximately 10,000 parameters. Compared to large-scale TS models with millions or billions of parameters, DynaMix is roughly 0.1% the size and offers inference times that are orders of magnitude faster. This suggests that encoding dynamical systems theory as a prior is significantly more efficient than raw scaling for DSR tasks.

4.2 Long-Term Accuracy and Generalization

DynaMix significantly outperformed foundation models like Chronos, Mamba4Cast, and TimesFM in terms of geometrical agreement (attractor shape) and temporal statistics (power spectrum). While other models often produced "meaningless dynamics" or flat lines when applied to human physiological data, DynaMix generated forecasts that maintained the statistical signatures of the original signals.

4.3 Zero-Shot Transfer

Despite being trained exclusively on 3D synthetic systems, DynaMix successfully reconstructed 2D and 6D systems. This demonstrates that the model learns transferable representations of motion and coupling that generalize across dimensions and domains.

5. Limitations and Future Work

While DynaMix represents a significant step forward, several limitations remain:

  • Stationarity: The model assumes the underlying rules of a system are constant. It struggles with "tipping points" or non-stationary data where parameters shift mid-stream.
  • Multi-scale Dynamics: Systems with widely differing time scales (mixing very fast and very slow components) remain a challenge.
  • Embedding Sensitivity: For 1D data, the quality of reconstruction is highly dependent on the selection of the correct embedding dimension.

Future research will focus on continuous-time versions of the experts to handle irregularly sampled data and the incorporation of explicit decomposition modules to manage non-stationary trends.

6. Conclusion

DynaMix proves that foundation models for dynamical systems do not require massive parameter counts if they are grounded in the mathematical principles of dynamics. By prioritizing the reconstruction of the system's "climate" over simple point forecasting, DynaMix offers a robust, efficient, and truly zero-shot solution for understanding complex systems across scientific disciplines.

Summary updated

BibTeX
@inproceedings{hemmer2025true,
  author = {Christoph Jürgen Hemmer and Daniel Durstewitz},
  title = {{True Zero-Shot Inference of Dynamical Systems Preserving Long-Term Statistics}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=RE97LT26w8},
}
Figure