SAOT: An Enhanced Locality-Aware Spectral Transformer for Solving PDEs

Chenhong Zhou, Jie Chen, Zaifeng Yang

Abstract

The Spectral Attention Operator Transformer (SAOT) is a novel neural operator designed to address the "over-smoothing" problem in Partial Differential Equation (PDE) solvers. By integrating the global reach of Fourier transforms with the local precision of Wavelet transforms through a gated fusion mechanism and a linear-complexity attention module, SAOT achieves state-of-the-art accuracy across diverse benchmarks while maintaining computational efficiency and discretization invariance.

Introduction

Partial Differential Equations (PDEs) are fundamental to physics and engineering, yet traditional numerical solvers are often computationally prohibitive. Neural Operators, such as the Fourier Neural Operator (FNO), offer a faster alternative but suffer from spectral bias, struggling to capture local, high-frequency details like sharp gradients or turbulence. While Transformers could potentially address these issues, their standard self-attention mechanism has a quadratic complexity ($O(N^2)$), making them impractical for the high-resolution meshes required in scientific computing. SAOT bridges this gap by capturing both global and local features with linear efficiency.

Methodology: The SAOT Architecture

The core of SAOT is the Spectral Attention layer, which utilizes a dual-pathway approach to process information across different scales.

The SAOT architecture featuring the dual-pathway Spectral Attention layer with Fourier and Wavelet components.

Wavelet Attention (WA)

The primary innovation of SAOT is Wavelet Attention. It employs a Fast Wavelet Transform (FWT) to decompose inputs into frequency subbands. A linearized, kernel-based attention mechanism is then applied to these subbands. Since wavelets are localized in both space and frequency, this module is uniquely equipped to capture fine-grained local details that Fourier-based methods often miss.

Fourier Attention (FA) and Gated Fusion

In parallel, the Fourier Attention module captures global dependencies across the entire domain. To combine these distinct perspectives, SAOT employs a Gated Fusion Block. This block uses a learnable, sigmoid-activated linear layer to adaptively weight the contributions of the Fourier and Wavelet pathways at every point in the mesh, ensuring the model can prioritize local or global information as needed by the underlying physics.

Efficiency and Complexity

By leveraging linearized attention and spectral transforms, SAOT maintains a complexity of $O(N \log N)$ or $O(N)$. This efficiency allows the model to scale effectively to high-resolution simulations without the memory bottlenecks of standard Transformers.

Experimental Evaluation

SAOT was evaluated on six major benchmarks spanning fluid and solid mechanics: Darcy Flow, Navier-Stokes (NS), Airfoil (structured mesh), Pipe flow, Plasticity (material deformation), and Elasticity (unstructured point clouds).

Key Results

SAOT achieved state-of-the-art (SOTA) performance across all tested benchmarks, consistently outperforming previous leading models like Transolver and FNO.

Comparison of SAOT against baseline models across six PDE benchmarks, showing relative L2 error.

Notable improvements include:

  • Navier-Stokes: A 22.3% reduction in relative $L^2$ error compared to the previous best (0.0985 to 0.0688).
  • Darcy Flow: Error reduction from 0.0058 to 0.0049.
  • Plasticity: Achieved a remarkably low error of 0.0008, significantly outperforming all baselines.
  • Complex Geometries: On the Elasticity benchmark involving unstructured point clouds, SAOT maintained a low error of 0.0080, demonstrating robustness where traditional spectral operators often fail.

Discussion and Insights

The success of SAOT stems from its ability to handle the complementary nature of Fourier and Wavelet domains. While Fourier methods are effective for global patterns, they lose spatial localization. Wavelets provide a "best of both worlds" decomposition, essential for PDEs where local shocks or boundary conditions dictate global behavior. Furthermore, SAOT exhibits discretization invariance; it can be trained on low-resolution meshes and applied to higher resolutions with minimal accuracy loss, a property known as "zero-shot super-resolution."

Conclusion and Future Work

SAOT represents a significant step forward in neural operators by effectively mitigating spectral bias through locality-aware attention. Future research may explore more sophisticated wavelet bases, such as Daubechies or Coiflets, and investigate the memory demands of scaling the architecture to 3D simulations.

Summary updated

BibTeX
@inproceedings{zhou2026saot,
  author = {Chenhong Zhou and Jie Chen and Zaifeng Yang},
  title = {{SAOT: An Enhanced Locality-Aware Spectral Transformer for Solving PDEs}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i34.40128},
}
Figure