Abstract
In this paper, we apply the self-attention from the state-of-the-art Transformer in Attention Is All You Need for the first time to a data-driven operator learning problem related to partial differential equations (PDEs). We demonstrate that the softmax normalization in scaled dot-product attention is sufficient but not necessary for operator approximation. By removing softmax, the attention mechanism's capacity becomes comparable to a Petrov-Galerkin projection. We introduce the Galerkin Transformer, which utilizes a new layer normalization scheme and linear complexity attention to achieve remarkable accuracy and efficiency in operator learning tasks.
Introduction: Bridging Transformers and PDEs
Operator learners, such as the Fourier Neural Operator (FNO), aim to learn mappings between infinite-dimensional function spaces—for instance, mapping physical coefficients to solution fields in PDEs. While Transformers excel in sequence modeling, their application to physical systems is hindered by the quadratic complexity of standard self-attention and the mathematical misalignment of softmax normalization with continuous function data.
This research addresses these hurdles by re-interpreting attention through the lens of operator approximation theory in Hilbert spaces. The authors argue that for physical operators, attention acts as a learnable projection, and softmax is computationally and mathematically redundant.
Methodology: The Galerkin Transformer (GT)
The Galerkin Transformer modifies the standard Transformer architecture to better suit the requirements of physical operator learning.
Key Architectural Innovations
- Attention without Softmax: By replacing softmax-normalized dot-product attention with a linear variant, the operation becomes equivalent to a Petrov-Galerkin projection. This allows the model to be interpreted as an integral transform, aligning it with classical numerical methods.
- Galerkin-type Layer Normalization: A specialized normalization scheme is introduced to allow learned scaling to propagate through the network, facilitating the handling of unnormalized physical data.
- Linear Complexity: By rearranging matrix multiplications—computing $(K^T V)$ before multiplying by $Q$—the computational complexity is reduced from $O(n^2)$ to $O(n)$, where $n$ is the number of grid points.
- Diagonal Dominant Initialization: Initializing projection matrices with a small identity component significantly stabilizes training, improving performance by up to 50%.
Experiments and Key Results
The GT was evaluated on three benchmarks: the 1D viscid Burgers’ equation, 2D interface Darcy flow, and an inverse interface coefficient identification problem.
Performance Gains
- Efficiency: As shown in the performance table, the Galerkin Transformer (GT) achieves the lowest memory usage and highest speed among compared variants. It offers a significant reduction in GFLOPs and memory footprint compared to standard Transformers (ST).
- Accuracy: GT outperformed state-of-the-art models like FNO on the Burgers’ equation.
- Robustness: In inverse problems, the model successfully recovered high-frequency interface traits from noisy data, a scenario where traditional smoothers often fail.
Conclusion and Future Directions
The Galerkin Transformer demonstrates that attention-based models can be both mathematically grounded and highly efficient for PDE solving. While currently limited to encoder-only architectures and requiring downsampling for extremely high resolutions, it represents a significant step forward in data-driven operator learning. Future work will explore extending these principles to causal architectures and more complex, non-smooth physical systems.
Summary updated