StFT: Spatio-temporal Fourier Transformer for Long-term Dynamics Prediction

Da Long, Shandian Zhe, Samuel Williams, Leonid Oliker, Zhe Bai

TL;DR

The Spatio-temporal Fourier Transformer (StFT) is a novel deep learning architecture designed for stable, long-term forecasting of complex multi-scale physical systems like plasma fusion and atmospheric dynamics. By combining a hierarchical coarse-to-fine spatial representation with a dual-path (frequency and spatio-temporal) transformer block, StFT accurately captures both global structures and fine-grained details. Its probabilistic variant, StFT-F, integrates a generative flow-matching mechanism to correct accumulated errors over time and provide well-calibrated uncertainty quantification.

What Problem Does This Paper Address?

Simulating physical systems governed by partial differential equations (PDEs)—such as turbulent fluids, weather patterns, or magnetically confined plasma—is notoriously difficult. These systems are inherently multi-scale and multi-physics, meaning that micro-scale turbulence directly influences macro-scale flows.

Traditional numerical solvers are highly accurate but computationally expensive and scale poorly. While machine learning models (like neural operators) have emerged as fast alternatives, they struggle with long-term forecasting. When rolled out autoregressively (using past predictions to forecast the next step), small errors compound rapidly, leading to unphysical artifacts, phase shifts, and total simulation breakdown. Furthermore, most existing neural operators lack built-in uncertainty quantification (UQ), making it impossible to know when a long-term prediction has drifted away from reality.

Methods

The authors introduce StFT and its probabilistic extension StFT-F, which tackle these challenges through three core architectural innovations:

  1. Hierarchical Scale Learning: Instead of processing the entire spatial domain at a single resolution, StFT uses a hierarchy of blocks. The first block looks at large, coarse patches to capture global, macroscopic structures. Subsequent blocks iteratively subdivide these patches into finer granularities to resolve localized, high-frequency details.
  2. Dual-Path Transformer Blocks: Within each hierarchical layer, the model splits feature processing into two parallel paths:
  3. Frequency Path: Uses a Fast Fourier Transform (FFT) and low-frequency filtering to mix global information efficiently.
  4. Spatio-temporal Path: Operates in standard physical space using self-attention to capture localized, fine-scale features.
  5. Overlapping Tokenizer: Standard vision-transformer-style patching causes discontinuous "blocky" artifacts at patch boundaries. StFT introduces an overlapping tokenizer where adjacent patches share boundary pixels, which are later averaged out during detokenization to ensure spatial smoothness.
  6. Generative Residual Correction (StFT-F): To prevent error accumulation during long rollouts, the authors append a generative rectified flow matching block. This block treats the deterministic StFT prediction as a baseline and models the remaining stochastic residual, allowing the network to sample multiple plausible future trajectories and quantify prediction uncertainty.
Figure 1: Left: Overview of the proposed StFT and StFT-F model. The model predicts ut+1 using the past k snapshots ˜ut, employing L spatial refinements from coarse to fine scales through the proposed Spatiotemporal Fourier Transformer (StFT) blocks. Bottom left: an illustration of the overlapping tokenizer, where the patch size is 2 × 2, and the overlapping number is 1 × 1. Right: Illustration of the proposed StFT block. First, the past snapshots ˜ut and a coarser prediction from the previous layer are temporally stacked. The stacked discretizations are passed to the overlapping tokenizer to generate tokens for each variable. Next, tokens corresponding to different variables in the same spatial are mixed through a variable mixer. Two paths of transformation, frequency path and spatiotemporal path, process the frequency embeddings and the spatiotemporal embeddings respectively. The finer prediction for timestamp t + 1 is obtained after passing through the overlapping detokenizer.

Datasets

The model was evaluated on three highly non-linear, multi-physics benchmark datasets:

  • Plasma MHD: A high-dimensional dataset simulating magnetohydrodynamic instabilities in tokamak fusion reactors based on real-world DIII-D experimental discharges (6 coupled variables, up to 244 timesteps).
  • 2D Incompressible Navier-Stokes: Classic fluid dynamics modeling velocity and pressure fields driven by an external force.
  • Spherical Shallow-Water Equations: A viscous fluid model on a sphere used to simulate large-scale atmospheric flows and planetary weather patterns.

Key Insights and Ideas

  • The Best of Both Worlds: Pure Fourier Neural Operators (FNOs) excel at global, smooth features but blur out sharp local gradients. Spatial transformers capture fine details but struggle with global context. StFT’s dual-path architecture successfully unifies them.
  • Overlapping Patches Matter Immensely: Simply allowing patches to overlap and averaging their boundaries acts as an incredibly powerful regularizer for physical continuity. In ablation studies, adding the overlapping tokenizer dropped the relative error in the Plasma MHD dataset by 68% without adding noticeable computational overhead.
  • Embracing the Residual: Instead of forcing the deterministic network to perfectly fit chaotic trajectories, StFT-F delegates the unmodeled sub-scale physics and noise to a generative flow-matching model. This keeps the rollout stable over twice the time horizon of standard baselines.
Figure 2: A patch is subdi- vided into smaller patches for a hierarchical learning.

Key Results

  • Superior Accuracy: StFT and StFT-F consistently outperformed state-of-the-art baselines (including AFNO/DPOT, FNO, ViT, and U-Net). On the complex Plasma MHD dataset, StFT achieved a 3x error reduction compared to the best baseline (AR-DPOT).
  • Long-Term Stability: While baseline models suffered from rapid phase degradation and unphysical drift after short time horizons, StFT accurately tracked dominant wave phases even past 60+ autoregressive steps.
  • Calibrated Uncertainty: StFT-F's empirical confidence intervals were exceptionally well-calibrated. For a target 90% confidence interval, the actual empirical coverage achieved was 89.4% for Plasma and 89.5% for Shallow-Water equations, providing reliable bounds on when the model's confidence deteriorates.
  • Computational Efficiency: Despite its multi-scale nature, StFT is highly efficient. For the plasma dataset, it achieved a 30% reduction in training time and a 50% reduction in inference time compared to standard FNOs.

Limitations and Future Work

  • Regular Grid Constraint: The current implementation relies on standard rectangular patching and FFTs, meaning it is restricted to regular grids. It cannot natively handle the complex, irregular geometries typical of real-world engineering components (like irregular turbine blades or specific fusion reactor walls).
  • Future Directions: The authors plan to extend the framework to irregular domains using geometric deep learning or graph-based tokenizers. They also highlight the potential for implementing model parallelism across the hierarchical blocks to scale up to even larger, high-resolution simulations.

Summary updated

Abstract

Simulating the long-term dynamics of multi-scale and multi-physics systems poses a significant challenge in understanding complex phenomena across science and engineering. The complexity arises from the intricate interactions between scales and the interplay of diverse physical processes, which manifest in PDEs through coupled, nonlinear terms that govern the evolution of multiple physical fields across scales. Neural operators have shown potential in short-term prediction of such complex spatio-temporal dynamics; however, achieving stable high-fidelity predictions and providing robust uncertainty quantification over extended time horizons remains an open and unsolved area of research. These limitations often lead to stability degradation with rapid error accumulation, particularly in long-term forecasting of systems characterized by multi-scale behaviors involving dynamics of different orders. To address these challenges, we propose an autoregressive Spatio-temporal Fourier Transformer (StFT), in which each transformer block is designed to learn the system dynamics at a distinct scale through a dual-path architecture that integrates frequency-domain and spatio-temporal representations. By leveraging a structured hierarchy of \ours blocks, the resulting model explicitly captures the underlying dynamics across both macro- and micro- spatial scales. Furthermore, a generative residual correction mechanism is introduced to learn a probabilistic refinement temporally while simultaneously quantifying prediction uncertainties, enhancing both the accuracy and reliability of long-term probabilistic forecasting. Evaluations conducted on three benchmark datasets (plasma, fluid, and atmospheric dynamics) demonstrate the advantages of our approach over state-of-the-art ML methods.
BibTeX
@article{long2025stft,
  author = {Da Long and Shandian Zhe and Samuel Williams and Leonid Oliker and Zhe Bai},
  title = {{StFT: Spatio-temporal Fourier Transformer for Long-term Dynamics Prediction}},
  journal = {Accepted by TMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=o9Cb0ri2oW},
}
Figure