Attention as Lattice Gauge Theory: Relations Interact Only Through Wilson Loops

Aakash Sarkar, Ananya Mukherjee, Motoshige SatoarXiv 2026 · arXiv

TL;DR

This paper proves that if we treat entities in a neural network as having independent coordinate frames, the resulting attention mechanism is mathematically forced into the form of Lattice Gauge Theory. This "Holonomy Attention" requires relations to interact only through closed loops (Wilson loops), providing a rigorous symmetry-based foundation for successful but previously heuristic designs like AlphaFold’s triangle updates.

What Problem Does This Paper Address?

In deep learning, we often design "relational" architectures—like Graph Neural Networks or Transformers—based on intuition. For example, AlphaFold 2 uses "triangle updates" to refine the relationships between amino acids. But why that specific shape? Is it just one good design among many, or is it fundamentally "correct"?

The authors tackle this by introducing the concept of per-entity frame freedom. Imagine a story with characters Alice, Bob, and Carol. Each character's internal representation (their "embedding") shouldn't depend on a global coordinate system. If you rotate Alice's coordinate frame, the model's understanding of her relations to others should remain consistent. The paper investigates what mathematical constraints this symmetry imposes on how relations interact.

Methods

The authors draw a direct map between attention and Lattice Gauge Theory (LGT), a framework from high-energy physics. In this dictionary:

  • Entities map to sites on a lattice.
  • Relations map to "link variables" ($T_{ij}$) that perform parallel transport between sites.
  • Interactions occur around "plaquettes" (the smallest closed loops).

They prove a "forced form" theorem: if you respect frame freedom, relations that don't share a cycle cannot interact. The only way to score a triangle of relations ($i, j, k$) in a gauge-invariant way is through its holonomy ($H = T_{ij}T_{jk}T_{ki}$)—specifically, the trace of this holonomy, known as a Wilson loop.

Figure 1: Holonomy attention, zero-shot. (a) The forced update. (b) Accuracy on unobserved S4 relations versus corrupted fraction (14 entities, degree 8, four rounds; mean and sd, 10 instances). (c) Holonomy score tr H/d of observed triangles at p=0.3 (discrete: permutation traces are integers). (d) The same score separates within- from cross-cluster triangles with no labels.

The authors instantiate this as Holonomy Attention, a zero-shot mechanism that updates edges by aggregating messages from neighboring triangles, weighted by their holonomy scores.

Datasets

The researchers evaluated their theory on group synchronization tasks—recovering frames from noisy pairwise relations—using the symmetric group $S_4$ (the 24 ways to permute four objects). They tested on:

  • Sparse and Dense Graphs: Randomly generated graphs where some relations are observed and others must be inferred.
  • Corrupted Data: Scenarios where a fraction $p$ of the observed relations are replaced with random noise.
  • Clustered Graphs: Graphs with two distinct communities to test unsupervised cluster separation without labels.

Key Insights and Ideas

  • The "No Cycle, No Interaction" Rule: Without a closed loop, there is no gauge-invariant way for two relations to "talk" to each other. This explains why "global edge attention" is physically inconsistent unless it operates over cycles.
  • Standard Attention is a Special Case: The authors show that standard dot-product attention is actually the "flat-connection limit" of this theory—a simplified version where all loops have trivial holonomy.
  • The Abelian Ceiling: Many existing knowledge graph models (like RotatE) are "abelian," meaning their operators commute. The authors prove these models are fundamentally capped at low accuracy (0.083 for $S_4$) on non-commutative tasks because they can't distinguish the order of relations (e.g., a father's mother vs. a mother's father).
  • Wilson Loops as Error Detectors: Because a "clean" triangle should have a consistent holonomy, the Wilson loop serves as a perfect, unsupervised diagnostic for corrupted data.

Key Results

  • Zero-Shot Performance: On sparse $S_4$ graphs with 20% corruption, Holonomy Attention achieved 0.88 accuracy in inferring unobserved relations without any training. Commutative models were stuck at 0.083.
  • Robustness: Even with 45% of edges corrupted, the model maintained significant predictive power, performing at or near the level of specialist synchronization algorithms.
  • Corruption Detection: The holonomy score flagged corrupted triangles with an AUC of 0.99, whereas commuting scores were capped at 0.76.
  • Unsupervised Clustering: The mechanism separated clusters in a graph with 0.98 AUC based purely on relational consistency, without needing any labels.

Limitations and Future Work

The current work uses a "pure-gauge" approach where relation operators are selected from a fixed library (the group $S_4$). A major next step is making the gauge field dynamical—allowing the model to generate these transport operators directly from the context of the data, rather than retrieving them from a library.

Additionally, while the paper provides the "edge-level" half of a complete architecture, fully integrating this with "node-level" matter fields (the actual entity vectors) is required to build a general-purpose "Holonomy Transformer." The authors suggest that AlphaFold's success is partly because its "pair stack" is effectively learning to flatten these connections, driving the geometry toward zero holonomy.

Summary updated

Abstract

Relations compose through the entity they share - Alice is Bob's mother and Bob is Carol's father, so Alice is Carol's grandmother - and the order matters, since a father's mother is not a mother's father; verifying a composed relation means closing the loop. We show that if each entity's coordinate frame may be rotated independently (per-entity frame freedom), three things follow. In the pure-gauge sector, relations that share no cycle cannot interact at all. The only gauge-invariant score of a triangle is a class function of its holonomy $H = T_{ij} T_{jk} T_{ki}$ - a Wilson loop. And the only equivariant update of an edge is $U = Ψ(H)\, T_{ij}$ with $Ψ$ conjugation-equivariant. Edge attention by Wilson loops is therefore not one design among many but the unique form the symmetry permits. Three communities hold pieces of this form: the commuting-operator embeddings of knowledge graphs are its abelian (commuting) sector; AlphaFold's triangle update and the Edge Transformer have its shape - an edge updated through the third vertex - without its constraints; and lattice QCD builds it equivariantly on lattices. We instantiate the equivariant sub-family on general relational graphs as holonomy attention. With no training and composition alone, it infers unobserved $S_4$ relations on sparse graphs at 0.88 with a fifth of the edges corrupted, at or slightly below specialist synchronization algorithms, where every commutative architecture is capped at 0.083; its holonomy score flags corrupted triangles at AUC 0.99 where commuting scores are capped at 0.76 by a second selection rule, and it separates clusters without labels.
BibTeX
@misc{sarkar2026attention,
  author = {Aakash Sarkar and Ananya Mukherjee and Motoshige Sato},
  title = {{Attention as Lattice Gauge Theory: Relations Interact Only Through Wilson Loops}},
  year = {2026},
  eprint = {2610.37656},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.37656},
}
Figure