Abstract
The Spectral Attention Operator Transformer (SAOT) is a novel neural operator designed to address the "over-smoothing" problem in Partial Differential Equation (PDE) solvers. By integrating the global reach of Fourier transforms with the local precision of Wavelet transforms through a gated fusion mechanism and a linear-complexity attention module, SAOT achieves state-of-the-art accuracy across diverse benchmarks while maintaining computational efficiency and discretization invariance.
Introduction
Partial Differential Equations (PDEs) are fundamental to physics and engineering, yet traditional numerical solvers are often computationally prohibitive. Neural Operators, such as the Fourier Neural Operator (FNO), offer a faster alternative but suffer from spectral bias, struggling to capture local, high-frequency details like sharp gradients or turbulence. While Transformers could potentially address these issues, their standard self-attention mechanism has a quadratic complexity ($O(N^2)$), making them impractical for the high-resolution meshes required in scientific computing. SAOT bridges this gap by capturing both global and local features with linear efficiency.
Methodology: The SAOT Architecture
The core of SAOT is the Spectral Attention layer, which utilizes a dual-pathway approach to process information across different scales.
Wavelet Attention (WA)
The primary innovation of SAOT is Wavelet Attention. It employs a Fast Wavelet Transform (FWT) to decompose inputs into frequency subbands. A linearized, kernel-based attention mechanism is then applied to these subbands. Since wavelets are localized in both space and frequency, this module is uniquely equipped to capture fine-grained local details that Fourier-based methods often miss.
Fourier Attention (FA) and Gated Fusion
In parallel, the Fourier Attention module captures global dependencies across the entire domain. To combine these distinct perspectives, SAOT employs a Gated Fusion Block. This block uses a learnable, sigmoid-activated linear layer to adaptively weight the contributions of the Fourier and Wavelet pathways at every point in the mesh, ensuring the model can prioritize local or global information as needed by the underlying physics.
Efficiency and Complexity
By leveraging linearized attention and spectral transforms, SAOT maintains a complexity of $O(N \log N)$ or $O(N)$. This efficiency allows the model to scale effectively to high-resolution simulations without the memory bottlenecks of standard Transformers.
Experimental Evaluation
SAOT was evaluated on six major benchmarks spanning fluid and solid mechanics: Darcy Flow, Navier-Stokes (NS), Airfoil (structured mesh), Pipe flow, Plasticity (material deformation), and Elasticity (unstructured point clouds).
Key Results
SAOT achieved state-of-the-art (SOTA) performance across all tested benchmarks, consistently outperforming previous leading models like Transolver and FNO.
Notable improvements include:
- Navier-Stokes: A 22.3% reduction in relative $L^2$ error compared to the previous best (0.0985 to 0.0688).
- Darcy Flow: Error reduction from 0.0058 to 0.0049.
- Plasticity: Achieved a remarkably low error of 0.0008, significantly outperforming all baselines.
- Complex Geometries: On the Elasticity benchmark involving unstructured point clouds, SAOT maintained a low error of 0.0080, demonstrating robustness where traditional spectral operators often fail.
Discussion and Insights
The success of SAOT stems from its ability to handle the complementary nature of Fourier and Wavelet domains. While Fourier methods are effective for global patterns, they lose spatial localization. Wavelets provide a "best of both worlds" decomposition, essential for PDEs where local shocks or boundary conditions dictate global behavior. Furthermore, SAOT exhibits discretization invariance; it can be trained on low-resolution meshes and applied to higher resolutions with minimal accuracy loss, a property known as "zero-shot super-resolution."
Conclusion and Future Work
SAOT represents a significant step forward in neural operators by effectively mitigating spectral bias through locality-aware attention. Future research may explore more sophisticated wavelet bases, such as Daubechies or Coiflets, and investigate the memory demands of scaling the architecture to 3D simulations.
Summary updated