3D Positional Encodings for Geometric Transformers

39 concepts33 papers1 proposal

Standard Transformer architectures are fundamentally blind to structure; they treat inputs as an unordered bag of tokens. In natural language processing, we fix this by adding 1D positional encodings. In computer vision, we use 2D grids. However, the physical world is three-dimensional. When transformers are applied to continuous geometric data—such as molecular structures, point clouds, RGB-D images, or physical sensor arrays—standard 1D or 2D encodings fail.

If 3D coordinates are simply appended as raw input features, the self-attention mechanism treats them like any other semantic attribute. It struggles to inherently grasp that physical proximity dictates interactions, or that a geometric structure remains identical regardless of how it is rotated or translated in space. To bridge this gap, a major research direction focuses on 3D positional encodings that embed geometric invariants and physical distances directly into the attention mechanism.

The Intuition: Baking Geometry into Attention

To understand why explicit 3D encodings work, consider how standard self-attention computes the relationship between two tokens, $i$ and $j$:

$$\text{Attention}(Q, K) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right)$$

Without spatial awareness, the dot product $q_i k_j^T$ only measures semantic similarity. If we want the model to prioritize physically close objects, we must condition this dot product on the 3D geometric relationship between token $i$ and token $j$.

There are two primary ways to achieve this: 3D Rotary Positional Encodings, which manipulate the orientation of features in a latent space, and Direct Distance Modulation, which injects physical metrics directly into the attention weights.

3D Rotary Positional Encodings (3D RoPE)

In 1D text processing, the Rotary Position Embedding (RoPE) rotates query and key vectors in 2D planes by an angle proportional to their absolute position. When the dot product is taken, the absolute positions cancel out, leaving a result that depends purely on the relative distance between the tokens.

To extend this elegance to three dimensions, researchers apply rotation matrices independently along the three spatial axes ($x, y, z$). As explained by Shuqi Lu et al. in More Space Is All You Need: Revisiting Molecular Representation Learning, this formulation captures pairwise directional and distance information in 3D space with linear complexity, making it highly suitable for transformer architectures.

By extending rotary embeddings to three dimensions using block-diagonal matrices, the relative distance and orientation between two points are mathematically encoded directly into the dot product of their transformed features. This allows the network to natively reason about 3D structural relationships.

Standard 1D RoPE:   [Token i] ---> Rotate by θ(position_i) ───┐
                                                                ├─> Dot Product encodes relative 1D distance
                    [Token j] ---> Rotate by θ(position_j) ───┘

Geometric 3D RoPE:  [Point i] ---> Rotate by Rx(x), Ry(y), Rz(z) ───┐
                                                                    ├─> Dot Product encodes relative 3D distance & orientation
                    [Point j] ---> Rotate by Rx(x), Ry(y), Rz(z) ───┘

This mathematical trick solves several critical problems across different domains:

Modulating Attention via Physical Distance

While 3D RoPE modifies the queries and keys before the dot product, an alternative approach modulates the attention matrix directly using explicit physical distances.

In molecular modeling, the physical distance between atoms determines chemical bonds and molecular properties. In Relative molecule self-attention transformer, Tomasz Danel uses radial basis functions to map continuous Euclidean distances into high-dimensional vectors, allowing the attention mechanism to learn complex, non-linear spatial interactions. This ensures that atoms that are distant in a molecule's 2D graph connectivity but close in 3D physical space can strongly attend to one another. Similarly, as highlighted in The Open Catalyst Challenge 2021: Competition Report, integrating 3D distances directly into the attention mechanism allows models to maintain structural awareness and achieve rotational invariance.

For irregular spatial data like point clouds, simple additive biases are often insufficient. Li Jiang et al. show in Point Transformer V2: Grouped Vector Attention and Partition-based Pooling that modulating relation vectors with a positional encoding multiplier (rather than an additive bias) enables the network to capture far more sophisticated geometric dependencies.

Abstracting Away Fixed Sensor Layouts

Beyond performance boosts, explicit 3D positional encodings offer a powerful structural advantage: they decouple the transformer architecture from fixed hardware configurations or sensor grids.

In fields like neural decoding, electrode layouts vary significantly between different subjects. Geeling Chau et al. solve this in Population Transformer: Learning Population-level Representations of Neural Activity by utilizing the actual 3D spatial coordinates of the sensors as positional encodings. Because the model processes signals based on their physical location in 3D space rather than a fixed index list, it can seamlessly generalize to entirely new subjects with different numbers or arrangements of sensors.

A similar principle applies to autonomous driving, where data arrives from a messy mix of cameras, LiDAR, and RADAR. Apoorv Singh notes in Transformer-Based Sensor Fusion for Autonomous Driving: A Survey that embedding explicit 3D spatial coordinates directly into tokens allows the attention mechanism to natively resolve spatial misalignments across entirely different sensor modalities.

Even in standard computer vision tasks, treating depth as a true third spatial coordinate yields better results than treating it as a passive input channel. Francesco Barbato et al. demonstrate in DepthFormer: Multimodal Positional Encodings and Cross-Input Attention for Transformer-Based Segmentation Networks that extending 2D positional encodings into a 3D coordinate space using depth values allows models to capture geometric relationships using fixed mathematical functions, achieving significant performance gains without adding learned parameters to the encoder. This is mirrored by Nian Liu et al. in VST++: Efficient and Stronger Visual Saliency Transformer, where mapping depth values to a discrete 3D coordinate range allows single-stream transformers to perceive physical geometry with minimal computational overhead.

Key Foundational Work

SuperFormer: Volumetric Transformer Architectures for MRI Super-Resolution (2022) — contributed 2 concept elements to this direction.
Lepard: Learning partial point cloud matching in rigid and deformable scenes (2021) — contributed 2 concept elements to this direction.
DepthFormer: Multimodal Positional Encodings and Cross-Input Attention for Transformer-Based Segmentation Networks (2022) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_3d_positional_encoding_4a21fc21,
  title = {3D positional encoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/3d-positional-encoding/rdr_4a21fc217478434bb951af987cfb1deb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_3d_positional_encoding_4a21fc21_v1,
  title = {3D positional encoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/3d-positional-encoding/rdr_4a21fc217478434bb951af987cfb1deb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake