Choose a Transformer: Fourier or Galerkin

Shuhao Cao

Introduction

Partial Differential Equations (PDEs) are fundamental to modeling complex physical phenomena, from climate dynamics to fluid mechanics. Traditionally, solving these equations requires computationally expensive numerical simulations, such as Finite Element Methods, for every new initial condition or parameter set.

Recently, "operator learners" like the Fourier Neural Operator (FNO) have emerged as a powerful alternative. Instead of solving a single instance of a PDE, these models learn the underlying mapping between infinite-dimensional function spaces—for example, mapping a physical coefficient directly to a resulting fluid flow. While standard Transformers have revolutionized sequence modeling in natural language processing, they were not originally designed for continuous physical systems. Their standard "softmax" attention mechanism carries a quadratic computational cost ($O(n^2)$) relative to sequence length and lacks a clear mathematical interpretation when applied to continuous functions in Hilbert spaces.

This paper introduces the Galerkin Transformer, a specialized architecture tailored for learning the operators behind PDEs. By replacing standard softmax normalization with a mathematically grounded "Galerkin-type" attention, the model achieves linear complexity, higher accuracy, and significantly lower memory usage compared to traditional Transformers and state-of-the-art operator learners.

Rethinking Attention for Physical Systems

The core innovation of this work lies in reinterpreting the Query, Key, and Value matrices. Instead of treating their columns as discrete "words" or tokens, the author treats them as samples of continuous functions. This perspective leads to two new attention mechanisms:

  1. Fourier-type Attention: This variant computes an integral transform with a learnable kernel. While it retains the $O(n^2)$ complexity of standard attention, it completely removes the softmax operation. This allows for a "Nyström-style" approximation of integral equations, making it more suitable for physical data.
  2. Galerkin-type Attention: By changing the order of matrix multiplication in the attention mechanism, this variant achieves linear complexity ($O(nd^2)$) relative to the sequence length. Mathematically, the author proves this operation is equivalent to a Petrov-Galerkin projection, a foundational technique in classical numerical analysis.
Architecture comparison between standard attention and the proposed Galerkin attention.

Key Architectural Changes: * Softmax-Free Design: The paper demonstrates that softmax is "sufficient but not necessary" for approximation capacity. Removing it not only accelerates computation but also stabilizes the model when handling physical data. * Simple Layer Normalization: A novel normalization scheme mimicking a Galerkin projection is introduced, allowing scaling to propagate through attention layers and helping the model process unnormalized physical data effectively. * Improved Initialization: Utilizing a "diagonally dominant" initialization for projection matrices boosts overall performance by up to 50%.

Key Insights and Theoretical Bridge

A major conceptual breakthrough in this paper is the re-evaluation of softmax. In natural language processing, softmax "convexifies" weights to blend word meanings. However, in physics, we operate within Hilbert spaces. The author shows that without softmax, the attention mechanism effectively becomes a learnable projection. This establishes a rigorous theoretical bridge between Deep Learning and classical Numerical Analysis (specifically Céa’s Lemma).

Furthermore, because the Galerkin variant scales linearly with the number of grid points ($n$), it can process much higher-resolution simulations than standard Transformers, overcoming a significant bottleneck in PDE modeling.

Empirical Results

The models were rigorously evaluated on three standard PDE benchmarks: the 1D Viscid Burgers’ Equation, 2D Interface Darcy Flow, and a challenging Inverse Interface Coefficient Identification problem (where the model must deduce physical material properties from noisy fluid flow measurements).

Performance comparison showing the Galerkin Transformer (GT) achieving the lowest memory usage and highest speed.

The results demonstrate significant improvements across multiple metrics:

  • Accuracy and Robustness: The Galerkin Transformer (GT) and Fourier Transformer (FT) achieved 30% to 50% better evaluation accuracy than the baseline FNO on Darcy flow tasks. The GT also proved remarkably capable of recovering hidden coefficients from noisy data, a scenario where traditional smoothers often fail.
  • Computational Efficiency: As shown in the performance benchmarks, the Fourier variant reduced memory usage by 40-50% compared to standard softmax attention (ST). The Galerkin variant (GT) was even more efficient, offering up to a 120% speed boost over other linear attention variants while maintaining an $O(nd^2)$ complexity.
  • Resolution Invariance: Similar to FNOs, these models are resolution-invariant. They can be trained on low-resolution data and evaluated on high-resolution grids without the need for retraining.

Limitations and Future Directions

Despite its advantages, the Galerkin Transformer has some limitations. Because the matrix multiplication order is swapped to achieve linear scaling, the model is inherently non-causal (Encoder-only). While excellent for "encoding" an entire physical state, it cannot be used for step-by-step autoregressive "decoding" like a GPT model. Additionally, applying these operators at full resolution for massive 2D or 3D grids remains computationally challenging, often necessitating downsampling or CNN-based pre-processors.

Future work will explore integrating feature maps that explicitly preserve specific physical structures, such as differentiability or strict energy conservation laws, further bridging the gap between deep learning and physics-based modeling.

Summary updated

Abstract

In this paper, we apply the self-attention from the state-of-the-art Transformer in Attention Is All You Need for the first time to a data-driven operator learning problem related to partial differential equations. An effort is put together to explain the heuristics of, and to improve the efficacy of the attention mechanism. By employing the operator approximation theory in Hilbert spaces, it is demonstrated for the first time that the softmax normalization in the scaled dot-product attention is sufficient but not necessary. Without softmax, the approximation capacity of a linearized Transformer variant can be proved to be comparable to a Petrov-Galerkin projection layer-wise, and the estimate is independent with respect to the sequence length. A new layer normalization scheme mimicking the Petrov-Galerkin projection is proposed to allow a scaling to propagate through attention layers, which helps the model achieve remarkable accuracy in operator learning tasks with unnormalized data. Finally, we present three operator learning experiments, including the viscid Burgers' equation, an interface Darcy flow, and an inverse interface coefficient identification problem. The newly proposed simple attention-based operator learner, Galerkin Transformer, shows significant improvements in both training cost and evaluation accuracy over its softmax-normalized counterparts.
BibTeX
@inproceedings{cao2021choose,
  author = {Shuhao Cao},
  title = {{Choose a Transformer: Fourier or Galerkin}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=ssohLcmn4-r},
}
Figure