Choose a Transformer: Fourier or Galerkin

Shuhao Cao

Abstract

In this paper, we apply the self-attention from the state-of-the-art Transformer in Attention Is All You Need for the first time to a data-driven operator learning problem related to partial differential equations (PDEs). We demonstrate that the softmax normalization in scaled dot-product attention is sufficient but not necessary for operator approximation. By removing softmax, the attention mechanism's capacity becomes comparable to a Petrov-Galerkin projection. We introduce the Galerkin Transformer, which utilizes a new layer normalization scheme and linear complexity attention to achieve remarkable accuracy and efficiency in operator learning tasks.

Introduction: Bridging Transformers and PDEs

Operator learners, such as the Fourier Neural Operator (FNO), aim to learn mappings between infinite-dimensional function spaces—for instance, mapping physical coefficients to solution fields in PDEs. While Transformers excel in sequence modeling, their application to physical systems is hindered by the quadratic complexity of standard self-attention and the mathematical misalignment of softmax normalization with continuous function data.

This research addresses these hurdles by re-interpreting attention through the lens of operator approximation theory in Hilbert spaces. The authors argue that for physical operators, attention acts as a learnable projection, and softmax is computationally and mathematically redundant.

Methodology: The Galerkin Transformer (GT)

The Galerkin Transformer modifies the standard Transformer architecture to better suit the requirements of physical operator learning.

Comparison of standard attention (a) and the proposed Galerkin Transformer attention (b).

Key Architectural Innovations

  1. Attention without Softmax: By replacing softmax-normalized dot-product attention with a linear variant, the operation becomes equivalent to a Petrov-Galerkin projection. This allows the model to be interpreted as an integral transform, aligning it with classical numerical methods.
  2. Galerkin-type Layer Normalization: A specialized normalization scheme is introduced to allow learned scaling to propagate through the network, facilitating the handling of unnormalized physical data.
  3. Linear Complexity: By rearranging matrix multiplications—computing $(K^T V)$ before multiplying by $Q$—the computational complexity is reduced from $O(n^2)$ to $O(n)$, where $n$ is the number of grid points.
  4. Diagonal Dominant Initialization: Initializing projection matrices with a small identity component significantly stabilizes training, improving performance by up to 50%.

Experiments and Key Results

The GT was evaluated on three benchmarks: the 1D viscid Burgers’ equation, 2D interface Darcy flow, and an inverse interface coefficient identification problem.

Performance comparison showing the efficiency of the Galerkin Transformer (GT) against other variants.

Performance Gains

  • Efficiency: As shown in the performance table, the Galerkin Transformer (GT) achieves the lowest memory usage and highest speed among compared variants. It offers a significant reduction in GFLOPs and memory footprint compared to standard Transformers (ST).
  • Accuracy: GT outperformed state-of-the-art models like FNO on the Burgers’ equation.
  • Robustness: In inverse problems, the model successfully recovered high-frequency interface traits from noisy data, a scenario where traditional smoothers often fail.

Conclusion and Future Directions

The Galerkin Transformer demonstrates that attention-based models can be both mathematically grounded and highly efficient for PDE solving. While currently limited to encoder-only architectures and requiring downsampling for extremely high resolutions, it represents a significant step forward in data-driven operator learning. Future work will explore extending these principles to causal architectures and more complex, non-smooth physical systems.

Summary updated

Abstract

In this paper, we apply the self-attention from the state-of-the-art Transformer in Attention Is All You Need for the first time to a data-driven operator learning problem related to partial differential equations. An effort is put together to explain the heuristics of, and to improve the efficacy of the attention mechanism. By employing the operator approximation theory in Hilbert spaces, it is demonstrated for the first time that the softmax normalization in the scaled dot-product attention is sufficient but not necessary. Without softmax, the approximation capacity of a linearized Transformer variant can be proved to be comparable to a Petrov-Galerkin projection layer-wise, and the estimate is independent with respect to the sequence length. A new layer normalization scheme mimicking the Petrov-Galerkin projection is proposed to allow a scaling to propagate through attention layers, which helps the model achieve remarkable accuracy in operator learning tasks with unnormalized data. Finally, we present three operator learning experiments, including the viscid Burgers' equation, an interface Darcy flow, and an inverse interface coefficient identification problem. The newly proposed simple attention-based operator learner, Galerkin Transformer, shows significant improvements in both training cost and evaluation accuracy over its softmax-normalized counterparts.
BibTeX
@inproceedings{cao2021choose,
  author = {Shuhao Cao},
  title = {{Choose a Transformer: Fourier or Galerkin}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=ssohLcmn4-r},
}
Figure