Spatio-temporal Fourier Transformer (StFT) for Long-term Dynamics Prediction

Da Long, Shandian Zhe, Samuel Williams, Leonid Oliker, Zhe Bai

Abstract

Simulating the long-term dynamics of multi-scale and multi-physics systems poses a significant challenge in understanding complex phenomena across science and engineering. The complexity arises from the intricate interactions between scales and the interplay of diverse physical processes. Neural operators have emerged as promising models for predicting such dynamics due to their flexibility and computational efficiency. However, they often fail to effectively capture multi-scale interactions or quantify the uncertainties inherent in the predictions. These limitations lead to rapid error accumulation, particularly in long-term forecasting of systems characterized by complex and coupled dynamics. To address these challenges, we propose a spatio-temporal Fourier transformer (StFT), in which each transformer block is designed to learn dynamics at a specific scale. By leveraging a structured hierarchy of StFT blocks, the model explicitly captures dynamics across both macro- and micro- spatial scales. Furthermore, a generative residual correction mechanism is integrated to estimate and mitigate predictive uncertainties, enhancing both the accuracy and reliability of long-term forecasts. Evaluations conducted on three benchmark datasets (plasma, fluid, and atmospheric dynamics) demonstrate the advantages of our approach over state-of-the-art ML methods.

Introduction

Simulating multi-scale physical systems—ranging from turbulent eddies in atmospheric storms to chaotic magnetic fields in fusion reactors—is computationally intensive. While traditional numerical solvers are often slow, neural operators have emerged as a faster alternative. However, these models frequently suffer from stability issues: small initial errors accumulate rapidly, leading to unphysical results over long horizons. Additionally, most existing models are deterministic, lacking the uncertainty quantification necessary for high-stakes applications like meteorology or fusion energy.

Methodology

The proposed Spatio-temporal Fourier Transformer (StFT) addresses these issues through a hierarchical architecture and a probabilistic refinement mechanism.

Dual-Path StFT Block

The core of the architecture is the StFT block, which processes data through two parallel paths to balance global stability and local detail:

  1. Frequency Path: Utilizes Fourier transforms to capture global, large-scale patterns by focusing on low-frequency components.
  2. Spatio-temporal Path: Employs standard transformer layers to capture local, fine-grained details in the physical space.
The StFT-F framework (left) and the internal StFT block architecture (right).

Hierarchical Refinement

StFT adopts a "coarse-to-fine" approach. Instead of learning all scales simultaneously, the model uses a hierarchy of blocks. Initial blocks predict the general "big picture," while subsequent blocks refine the prediction at increasingly smaller spatial scales. An overlapping tokenizer is used to ensure spatial continuity and prevent edge artifacts between patches.

Hierarchical patch subdivision strategy for coarse-to-fine spatial refinement.

Generative Residual Correction (StFT-F)

To enhance long-term stability, the authors introduce StFT-F. This variant uses Flow Matching to learn the distribution of errors (residuals) left by the deterministic StFT model. This allows the system to "correct" its trajectory and provide probabilistic forecasts, offering a range of possible outcomes and associated uncertainty estimates.

Experimental Setup

The model was evaluated on three challenging benchmarks representing diverse physical domains:

  • Plasma MHD: Magnetohydrodynamics in a tokamak, using experimental data from the DIII-D facility.
  • 2D Navier-Stokes: A standard fluid dynamics benchmark involving velocity and pressure fields.
  • Spherical Shallow-Water: Large-scale atmospheric flow modeling on a rotating sphere, essential for weather forecasting.

Results and Discussion

The StFT framework demonstrates significant improvements over state-of-the-art methods:

  • Accuracy: In Plasma MHD simulations, StFT reduced prediction error by a factor of three compared to baselines like AR-DPOT and AFNO.
  • Stability: StFT-F maintained accurate predictions for at least twice as long as models like FNO or ViT for a given error threshold.
  • Efficiency: StFT is notably efficient; in the plasma dataset, it was 20x smaller and 8x faster (in terms of FLOPs) than the Fourier Neural Operator (FNO).
  • Uncertainty Quantification: The StFT-F variant provides well-calibrated uncertainty estimates, with ground truth falling within the 90% confidence interval approximately 89.4% of the time.

The "Dual Path" design proved essential: the frequency domain maintains long-term stability, while the spatial domain captures sharp physical transitions. Furthermore, the hierarchical structure allows the model to adapt to the specific multi-scale requirements of different physical systems.

Conclusion

StFT provides a robust and efficient framework for long-term dynamics prediction in complex physical systems. While currently optimized for regular grids, future work will focus on extending the architecture to irregular geometries common in engineering. Additionally, model parallelism across multiple GPUs could further scale the system for higher-resolution simulations.

Summary updated

Abstract

Simulating the long-term dynamics of multi-scale and multi-physics systems poses a significant challenge in understanding complex phenomena across science and engineering. The complexity arises from the intricate interactions between scales and the interplay of diverse physical processes. Neural operators have emerged as promising models for predicting such dynamics due to their flexibility and computational efficiency. However, they often fail to effectively capture multi-scale interactions or quantify the uncertainties inherent in the predictions. These limitations lead to rapid error accumulation, particularly in long-term forecasting of systems characterized by complex and coupled dynamics. To address these challenges, we propose a spatio-temporal Fourier transformer (StFT), in which each transformer block is designed to learn dynamics at a specific scale. By leveraging a structured hierarchy of StFT blocks, the model explicitly captures dynamics across both macro- and micro- spatial scales. Furthermore, a generative residual correction mechanism is integrated to estimate and mitigate predictive uncertainties, enhancing both the accuracy and reliability of long-term forecasts. Evaluations conducted on three benchmark datasets (plasma, fluid, and atmospheric dynamics) demonstrate the advantages of our approach over state-of-the-art ML methods.
BibTeX
@article{long2025spatio,
  author = {Da Long and Shandian Zhe and Samuel Williams and Leonid Oliker and Zhe Bai},
  title = {{Spatio-temporal Fourier Transformer (StFT) for Long-term Dynamics Prediction}},
  journal = {Accepted by TMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=o9Cb0ri2oW},
}
Figure