Symmetry Preserving Attention Mechanisms

84 concepts53 papers2 proposals

Standard self-attention is fundamentally permutation-invariant: it treats its inputs as an unordered bag of tokens. To make transformers aware of structure—such as the sequential nature of text or the spatial layout of an image—we typically inject absolute positional encodings.

However, absolute coordinates break geometric symmetries. If an object shifts or rotates within an image, its absolute coordinates change entirely, forcing the network to relearn the same feature at every possible position and orientation. To solve this, a rapidly growing body of research focuses on symmetry-preserving attention mechanisms. By constraining or reformulating the attention operator, these architectures ensure equivariance (where transforming the input transforms the internal features predictably) or invariance (where the output remains unchanged).

From Absolute to Relative: Achieving Translation Equivariance

The simplest geometric symmetry is translation: if an image shifts by ten pixels, its representation should shift by ten pixels rather than changing completely.

Standard softmax attention breaks this because absolute position embeddings anchor tokens to fixed coordinates. To restore translation equivariance, researchers replace absolute coordinates with relative offsets. As proven by Jean-Baptiste Cordonnier et al. in On the Relationship between Self-Attention and Convolutional Layers, multi-head self-attention with relative positional encodings is a mathematical superset of convolution. It allows the model to compute attention weights based purely on the distance between elements rather than their absolute locations.

This insight underpins architectures like the stand-alone self-attention models proposed by Niki Parmar et al. in Stand-Alone Self-Attention in Vision Models, where local relative positioning allows transformers to replace convolutions entirely while maintaining translation equivariance. Similarly, Hehe Fan et al. introduce Translution: Unifying Self-Attention and Convolution for Adaptive and Relative Modelling, which swaps absolute embeddings for relative offset-specific weight matrices to guarantee that the model remains robust when objects shift within a frame.

However, standard softmax attention can still introduce non-linear spatial dependencies that break fractional or sub-pixel translation invariance. To achieve perfect shift consistency, Hagay Michaeli and Daniel Soudry in Alias-Free ViT: Fractional Shift Invariance via Linear Attention replace softmax attention with linear cross-covariance attention. This computes interactions across channels rather than spatial tokens, preserving shift-equivariance natively.

Discrete Group Equivariance: Rotations and Reflections

While translation handles shifts, real-world data often exhibits discrete rotational and reflectional symmetries (the $E(2)$ group). If you rotate an input image by 90 degrees, a standard attention map does not automatically rotate with it; the symmetry breaks because the attention weights are not bound to the transformation group.

To enforce discrete group equivariance, the attention operator itself must be restructured. David W. Romero and Mark Hoogendoorn demonstrate in Co-Attentive Equivariant Neural Networks: Focusing Equivariance On Transformations Co-Occurring in Data that constraining attention weight matrices to be circulant—where each row is a cyclic shift of the previous one—forces the attention mask to shift predictably alongside input transformations.

Furthermore, true group-equivariant attention requires expanding the domain of the attention function. Instead of computing attention purely over spatial coordinates, models must compute dependencies across a joint space of spatial coordinates and discrete group elements (such as explicit rotation angles). As detailed in $E(2)$-Equivariant Vision Transformer by Renjun Xu et al., reformulating relative positional encodings to transform correctly with the coordinate system ensures that attention scores remain invariant under group transformations. This allows internal feature maps to track orientation changes perfectly.

To generate these representations, Sheir A. Zaheer et al. in REViT: Roto-reflection Equivariant Convolutional Vision Transformer utilize group-equivariant convolutions to construct the Query, Key, and Value matrices directly, preserving symmetry across both spatial and discrete group dimensions.

Continuous Symmetries and Physical Spaces

In physical simulations, 3D computer vision, and robotics, symmetries are rarely discrete; they operate on continuous manifolds like the 3D Euclidean group $E(3)$ or special Euclidean group $SE(3)$ (continuous rotations and translations).

Interestingly, standard transformer self-attention layers inherently possess a form of continuous rotation symmetry. As shown by Binchi Zhang and Zaiyi Zheng in Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion, applying orthogonal transformations to the Query and Key matrices leaves the attention matrix completely invariant. This continuous symmetry provides an infinite manifold of weight configurations that yield identical outputs, which can be leveraged to align independently trained models.

To explicitly harness this for 3D geometric data, researchers embed invariant geometric operations directly into the attention mechanism. In Geometric Algebra Transformer, Johann Brehmer et al. replace standard dot products with the invariant inner product of projective geometric algebra, ensuring full $E(3)$-equivariance across large-scale physical datasets.

For volumetric data, Soumyabrata Kundu in Steerable Transformers for Volumetric Data redefines the attention mechanism to operate on complex-valued Fourier components, utilizing positional encodings modulated by spherical harmonics to guarantee continuous $SE(d)$ equivariance. By ensuring that the relationships between distant tokens transform predictably under global coordinate shifts, these models maintain strict geometric consistency regardless of the data's orientation.

Key Foundational Work

Co-Attentive Equivariant Neural Networks: Focusing Equivariance On Transformations Co-Occurring in Data (2020) — contributed 5 concept elements to this direction.
Co-Attentive Equivariant Neural Networks: Focusing Equivariance On Transformations Co-Occurring In Data (2019) — contributed 3 concept elements to this direction.
Stand-Alone Self-Attention in Vision Models (2019) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_symmetry_preserving_attention_da27474f,
  title = {symmetry-preserving attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/symmetry-preserving-attention/rdr_da27474f68344c2dba713e8979fd37fa},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_symmetry_preserving_attention_da27474f_v1,
  title = {symmetry-preserving attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/symmetry-preserving-attention/rdr_da27474f68344c2dba713e8979fd37fa/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake