Relative Spatial Encoding for Coordinate Invariant Representations

23 concepts21 papers1 proposal

When designing neural networks to reason about physical space—whether segmenting an organ in a medical scan, navigating a robot through a room, or placing components on a microchip—our instinct is often to assign every element a position on a global coordinate grid. However, absolute coordinate systems introduce a fundamental flaw: they tie the network’s understanding to a specific, arbitrary origin and orientation. If a scene shifts, rotates, or scales, its absolute coordinates change entirely, forcing the model to relearn the same geometric structures at every possible location.

To build models that generalize effortlessly across space, a growing body of research shifts the paradigm from absolute positioning to relative spatial encoding. By representing space intrinsically through the geometric relationships between elements rather than their coordinates on a global axis, these architectures achieve mathematical invariance to translation, rotation, and scale.

The Problem of Absolute Coordinate Leakage

In standard convolutional and transformer architectures, spatial awareness is typically injected via absolute positional encodings. While this tells the network where something is in the frame, it frequently compromises the network's ability to generalize.

As demonstrated by Courtois et al. in Investigating Neural Architectures by Synthetic Dataset Design, conventional positional encodings inadvertently leak absolute coordinate information into the network's feature logic. This leakage breaks translation covariance, causing models to overfit to specific spatial positions and fail when inputs are slightly shifted. For instance, an object detector might excel at identifying a target in the center of an image but fail near the borders because it relies on absolute spatial anchors like zero-padding or frame boundaries.

To resolve this, the core intuition of relative spatial encoding is simple: force the model to be mathematically blind to absolute coordinates. As explored by Huang et al. in Practical Equivariances via Relational Conditional Neural Processes, replacing absolute coordinates with pairwise comparisons or relative offsets ensures that the model cannot perceive global shifts or rotations. Because the input representation itself remains identical regardless of global translation, invariance is baked directly into the architecture as a strict inductive bias.

Absolute Encoding (Vulnerable to shifts):
[Object A: (x=12, y=15)] ---> [Model] ---> Rigidly tied to origin (0,0)

Relative Encoding (Coordinate-Invariant):
[Object A: 5 units away from Anchor B, rotated 45°] ---> [Model] ---> Invariant to global shifts

Mechanisms of Relative Spatial Encoding

How do models compute and utilize these coordinate-invariant representations in practice? Researchers have converged on three primary strategies:

1. Pairwise Distances and Displacements

Instead of mapping a point to an axis, models can compute the exact geometric relationships between all pairs of elements. In transformer-like architectures, this relational data is injected directly into the attention or value calculation.

For example, Choukroun et al. utilize inter-element Euclidean distances within a bimodal transformer in Reconstructing the Hemodynamic Response Function via a Bimodal Transformer, ensuring that the learned representations of vascular and neural fields remain invariant to the orientation or translation of the input field. Similarly, in PointListNet: Deep Learning on 3D Point Lists, Fan et al. integrate both one-dimensional step distances and three-dimensional vector displacements directly into the value transformation of a transformer network, allowing robust classification of complex 3D structures like proteins irrespective of their global spatial orientation.

2. Centroid and Anchor-Based Reference Frames

Computing all-to-all pairwise relationships can be computationally expensive. An alternative approach establishes local coordinate systems centered around dynamically predicted or predefined reference points, such as centroids or anchors.

  • Anatomical Centroids: In Deep neural networks for anatomical brain segmentation, de Brébisson et al. calculate the Euclidean distance from a target voxel to the estimated centers of all predefined anatomical regions. This creates a relative coordinate system that bypasses the need for computationally heavy non-linear image registration.
  • Part Centers: To prevent a generative model from exploiting edge biases, He et al. introduce a relative positional encoding scheme in GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation that defines pixel locations relative to predicted object part centers, enforcing strict translational equivariance.
  • Environmental Anchors: For higher-level reasoning, Pagani et al. show in 3D Spatial Understanding in MLLMs: Disambiguation and Evaluation that encoding the geometric relationships between a target object, its distractors, and stable reference anchors allows multimodal large language models to navigate complex 3D environments far more effectively than raw numerical coordinates.

3. Relational Graphs and Structural Grammars

When the absolute scale or global orientation of an environment varies wildly, space can be modeled as a topological graph where edges represent relative distances, orientations, or poses.

This structural approach allows models to capture the "grammar" of a scene. In Mining viewpoint patterns in image databases, Hsu et al. model spatial layouts using relative distances and orientations between objects, discovering recurring scene arrangements that are invariant to the global camera position. For physical object manipulation, Johns et al. demonstrate in SceneScore: Learning a Cost Function for Object Arrangement that evaluating scenes through relative poses between objects ensures that learned arrangement patterns remain valid no matter where the entire group of objects is placed on a surface.

Why Relative Encoding Generalizes

The primary reason relative spatial encoding works so well across diverse domains is that it aligns the model's inductive bias with the underlying physics of the problem.

Consider the task of tracking trajectories or placing components. In TransferTraj: A Vehicle Trajectory Learning Model for Region and Task Transferability, Wei et al. utilize learnable rotation matrices to encode relative distances between sequential trajectory points. Because the model evaluates the geometric relationship between steps rather than absolute geographic coordinates, it achieves zero-shot transferability across entirely different cities.

A similar principle applies to macro-scale engineering. In TransPlace: Transferable Circuit Global Placement via Graph Neural Network, Ye et al. model circuit components using relative positions within a directed acyclic graph. This representation is invariant to Euclidean transformations like rotation and translation, allowing the model to generalize its placement strategies across completely different chip sizes and orientations.

By discarding the global crutch of absolute coordinates and forcing neural architectures to compute space relationally, these methods achieve a level of structural robustness that mimics human spatial awareness: recognizing a pattern not by where it sits in the universe, but by how its pieces fit together.

Key Foundational Work

Mining viewpoint patterns in image databases (2003) — contributed 1 concept elements to this direction.
Combining Structural Descriptions and Image-based Representations for Image, Object, and Scene Recognition (2005) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_relative_spatial_encoding_coordinate_inv_a0fe1941,
  title = {relative spatial encoding (coordinate-invariant representation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relative-spatial-encoding-coordinate-invariant-representation/rdr_a0fe194114ea4c86814e9eb5f1253dc3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_relative_spatial_encoding_coordinate_inv_a0fe1941_v1,
  title = {relative spatial encoding (coordinate-invariant representation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relative-spatial-encoding-coordinate-invariant-representation/rdr_a0fe194114ea4c86814e9eb5f1253dc3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake