SAOT: An Enhanced Locality-Aware Spectral Transformer for Solving PDEs

Chenhong Zhou, Jie Chen, Zaifeng Yang

TL;DR

The Spectral Attention Operator Transformer (SAOT) is a new hybrid architecture that solves Partial Differential Equations (PDEs) by combining the global reach of Fourier transforms with the local precision of Wavelet transforms. By using a gated fusion mechanism and linearized attention, SAOT achieves state-of-the-art accuracy across six major benchmarks while remaining computationally efficient and resolution-independent.

What Problem Does This Paper Address?

Solving PDEs is fundamental to science and engineering, from predicting weather to designing aircraft. Traditional numerical solvers are accurate but slow. Recently, Neural Operators (like the Fourier Neural Operator, or FNO) have emerged as lightning-fast alternatives that learn to map input physical states to output solutions.

However, FNOs have a "blurry vision" problem: they excel at capturing global patterns but struggle with local details and high-frequency "sharp" features (a phenomenon known as spectral bias). On the other hand, standard Transformers are great at capturing relationships but suffer from quadratic complexity—meaning if you double the resolution of your simulation mesh, the computational cost quadruples, making them impractical for high-resolution physics.

Methods

The authors introduce SAOT, which replaces standard self-attention with a dual-path Spectral Attention layer:

  1. Wavelet Attention (WA): This is the paper's core innovation. It uses the Fast Wavelet Transform (FWT) to decompose the input into different frequency subbands that retain spatial information. It then applies a linearized attention mechanism (using a kernel feature map to avoid the $O(N^2)$ bottleneck) to learn local, high-frequency features efficiently.
  2. Fourier Attention (FA): Running in parallel, this module uses the Fourier domain to capture long-range, global dependencies across the entire domain.
  3. Gated Fusion Block: Since some parts of a physical system need global context (like a pressure field) and others need local detail (like a shockwave), a learnable "gate" adaptively blends the outputs of WA and FA.

The entire model follows an Encoder-Processor-Decoder pipeline, where the Processor consists of multiple stacked SAOT blocks.

Datasets

The model was evaluated on six diverse operator learning benchmarks:

  • Regular Grids: Darcy Flow (fluid pressure) and Navier-Stokes (fluid velocity over time).
  • Structured/Irregular Meshes: Airfoil (aerodynamics), Pipe (fluid flow), and Plasticity (material deformation).
  • Point Clouds: Elasticity (internal stress in complex structures).

These datasets represent a mix of 2D and 3D physical systems, testing the model's ability to handle both simple grids and complex, non-uniform geometries.

Key Insights and Ideas

  • Wavelets as the Missing Link: Unlike Fourier transforms, which lose spatial localization, Wavelets are localized in both space and frequency. This makes them mathematically ideal for "zooming in" on discontinuities or abrupt changes in a physical field.
  • Linear Complexity is King: By linearizing the attention mechanism within the wavelet subbands, the authors keep the complexity at $O(N \log N)$ or $O(N)$, allowing the model to scale to massive meshes that would crash a standard Transformer.
  • The Energy Spectrum Proof: One of the most compelling "aha" moments in the paper is Figure 2(d). It shows that while Fourier-based models "drop off" and fail to predict high-frequency energy, the Wavelet path tracks the ground truth almost perfectly, explaining why SAOT produces sharper, more accurate solutions.

Key Results

SAOT established a new state-of-the-art (SOTA) across all six benchmarks, often by a significant margin:

  • Accuracy: On the Darcy Flow and Navier-Stokes tasks, SAOT reduced error by 15.5% and 22.3% respectively compared to the previous SOTA (Transolver).
  • Complex Geometry: In the Elasticity benchmark (unstructured point clouds), SAOT achieved a relative $L^2$ error of 0.80%, while most spectral-based competitors struggled to stay below 2%.
  • Discretization Invariance: The model exhibits "zero-shot super-resolution." You can train it on a low-res $85 \times 85$ grid and test it on a high-res $421 \times 421$ grid; SAOT maintains lower error rates across these resolution shifts than FNO or other Transformer variants.

Limitations and Future Work

While SAOT is highly effective, the authors note a few areas for growth:

  • Wavelet Selection: This work primarily uses the Haar wavelet for simplicity. Exploring more sophisticated wavelet bases (like Daubechies) might further improve performance for specific physical phenomena.
  • Computational Overhead: Although the complexity is linear/quasi-linear, the dual-path architecture (WA + FA) naturally uses more parameters and memory than a single-path model.
  • Time-Stepping: While tested on time-dependent problems like Navier-Stokes, the model currently treats time slices as part of the input/output. Future iterations could integrate more explicit temporal modeling (like recurrent structures) for long-term stability.

Summary updated

Abstract

Neural operators have shown great potential in solving a family of Partial Differential Equations (PDEs) by modeling the mappings between input and output functions. Fourier Neural Operator (FNO) implements global convolutions via parameterizing the integral operators in Fourier space. However, it often results in over-smoothing solutions and fails to capture local details and high-frequency components. To address these limitations, we investigate incorporating the spatial-frequency localization property of Wavelet transforms into the Transformer architecture. We propose a novel Wavelet Attention (WA) module with linear computational complexity to efficiently learn locality-aware features. Building upon WA, we further develop the Spectral Attention Operator Transformer (SAOT), a hybrid spectral Transformer framework that integrates WA’s localized focus with the global receptive field of Fourier-based Attention (FA) through a gated fusion block. Experimental results demonstrate that WA significantly mitigates the limitations of FA and outperforms existing Wavelet-based neural operators by a large margin. By integrating the locality-aware and global spectral representations, SAOT achieves state-of-the-art performance on six operator learning benchmarks and exhibits strong discretization-invariant ability.
BibTeX
@inproceedings{zhou2026saot,
  author = {Chenhong Zhou and Jie Chen and Zaifeng Yang},
  title = {{SAOT: An Enhanced Locality-Aware Spectral Transformer for Solving PDEs}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i34.40128},
}
Figure