GIST: Gauge-Invariant Spectral Transformers for Scalable Graph Neural Operators

Mattia Rigotti, Nicholas Thumiger, Thomas W. Frick

Abstract

Adapting transformer positional encoding to meshes and graph-structured data presents significant computational challenges: exact spectral methods require cubic-complexity eigendecomposition and can inadvertently break gauge invariance through numerical solver artifacts, while efficient approximate methods sacrifice gauge symmetry by design. Both failure modes cause catastrophic generalization in inductive learning, where models trained with one set of numerical choices fail when encountering different spectral decompositions of similar graphs or discretizations of the same mesh. We propose GIST (Gauge-Invariant Spectral Transformers), a new graph transformer architecture that resolves this challenge by achieving end-to-end $\mathcal{O}(N)$ complexity through random projections while algorithmically preserving gauge invariance via inner-product-based attention on the projected embeddings. We prove GIST achieves discretization-invariant learning with bounded mismatch error, enabling parameter transfer across arbitrary mesh resolutions for neural operator applications. Empirically, GIST matches state-of-the-art on standard graph benchmarks (e.g., achieving 99.50% micro-F1 on PPI) while uniquely scaling to mesh-based Neural Operator benchmarks with up to 750K nodes, achieving state-of-the-art aerodynamic prediction on the challenging DrivAerNet and DrivAerNet++ datasets.

Introduction

Graph Transformers have emerged as a powerful tool for modeling complex relational data, yet they face a fundamental "scalability vs. stability" trade-off. To provide models with structural context, researchers often use "spectral positional encodings" derived from the eigenvectors of the graph Laplacian. However, this approach is hindered by two major barriers. First, the computational cost of exact eigendecomposition scales at $\mathcal{O}(N^3)$, making it impractical for large-scale meshes used in engineering and physics. Second, eigenvectors are not unique; numerical solvers make arbitrary "gauge choices" (such as sign flips) that standard neural networks cannot inherently ignore. This lack of gauge invariance leads to catastrophic failure when a model trained on one discretization of a mesh is tested on another.

Methods: Scalable and Invariant Architecture

GIST (Gauge-Invariant Spectral Transformers) resolves these issues by combining efficient random projections with a symmetry-preserving attention mechanism. Instead of exact eigendecomposition, GIST utilizes FastRP (Fast Random Projections) to generate approximate spectral embeddings in linear $\mathcal{O}(N)$ time.

To ensure stability across different numerical choices, GIST employs Inner-Product Attention. By restricting the transformer to operate only on the inner products between embeddings, the model focuses on the underlying geometry (distances and angles) rather than the specific coordinates of the spectral space. This design choice, supported by the Johnson-Lindenstrauss lemma, ensures the model remains gauge-invariant by construction.

The GIST architecture features a three-branch structure (features, local, and global) with a specialized Gauge-Invariant Spectral Self-Attention mechanism.

As shown in the architecture diagram above, GIST utilizes a multi-scale approach inspired by EfficientViT, featuring three parallel branches:

  1. Feature Branch: Processes node features using linear attention.
  2. Local Branch: Captures immediate neighborhood information via graph convolutions.
  3. Global Branch: Leverages the new Gauge-Invariant and Gauge-Equivariant spectral attention to model long-range dependencies across the entire graph.

Experimental Results

The efficacy of GIST was evaluated across standard graph benchmarks and high-fidelity neural operator tasks. On transductive and inductive benchmarks like Cora, CiteSeer, and PPI, GIST matches or exceeds state-of-the-art performance, notably achieving a 99.50% micro-F1 on PPI.

The "Smoking Gun" Test for Gauge Invariance

A critical contribution of this work is the demonstration of discretization-invariant learning. In a controlled experiment, the authors trained GIST on a 50% decimated (coarse) mesh and tested it on a full-resolution (fine) mesh. While a standard non-gauge-invariant baseline saw its performance plummet to an $R^2$ of 0.526, GIST maintained a robust $R^2$ of 0.840, proving that algorithmic gauge invariance is essential for cross-resolution transfer.

GIST maintains high performance when transferring from coarse training meshes to fine test meshes, whereas non-invariant models fail.

Large-Scale Aerodynamics

GIST scales linearly to handle meshes with up to 750,000 nodes, a feat previously impossible for spectral methods without significant downsampling. On the DrivAerNet and DrivAerNet++ benchmarks—which involve predicting aerodynamic pressure on complex car geometries—GIST achieved state-of-the-art results, outperforming specialized models like Transolver and TripNet with a 20.10% relative $\ell_2$ error.

Conclusion

GIST provides a theoretically grounded and computationally efficient framework for applying spectral transformers to large-scale graph and mesh data. By recovering algorithmic symmetry through inner-product attention, it enables robust generalization across varying mesh resolutions. Future work will explore the application of GIST to more complex, time-dependent physics simulations and the integration of alternative kernel functions to further enhance its representational power.

Summary updated

Abstract

Adapting transformer positional encoding to meshes and graph-structured data presents significant computational challenges: exact spectral methods require cubic-complexity eigendecomposition and can inadvertently break gauge invariance through numerical solver artifacts, while efficient approximate methods sacrifice gauge symmetry by design. Both failure modes cause catastrophic generalization in inductive learning, where models trained with one set of numerical choices fail when encountering different spectral decompositions of similar graphs or discretizations of the same mesh. We propose GIST (Gauge-Invariant Spectral Transformers), a new graph transformer architecture that resolves this challenge by achieving end-to-end $\mathcal{O}(N)$ complexity through random projections while algorithmically preserving gauge invariance via inner-product-based attention on the projected embeddings. We prove GIST achieves discretization-invariant learning with bounded mismatch error, enabling parameter transfer across arbitrary mesh resolutions for neural operator applications. Empirically, GIST matches state-of-the-art on standard graph benchmarks (e.g., achieving 99.50% micro-F1 on PPI) while uniquely scaling to mesh-based Neural Operator benchmarks with up to 750K nodes, achieving state-of-the-art aerodynamic prediction on the challenging DrivAerNet and DrivAerNet++ datasets.
BibTeX
@misc{rigotti2026gist,
  author = {Mattia Rigotti and Nicholas Thumiger and Thomas W. Frick},
  title = {{GIST: Gauge-Invariant Spectral Transformers for Scalable Graph Neural Operators}},
  year = {2026},
  eprint = {2603.16849},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.16849},
}
Figure