Relative Distance Embeddings for Relational Text Modeling

176 concepts140 papers2 proposals

When humans read a sentence to extract relationships between entities, we naturally focus on the local context surrounding those entities. For instance, in the sentence "The company, which was founded in a small garage decades ago, acquired the startup," the relationship between company and startup is heavily signaled by the word acquired. Crucially, the importance of acquired is not determined by its absolute position in the sentence (e.g., being the 12th word), but rather by its relative proximity to the two target entities.

Standard neural network architectures often struggle with this intuition. Some are entirely position-blind, while others rely on absolute position embeddings that assign a fixed coordinate to each index in a sequence. This absolute framing is fragile: if an introductory clause is added to the beginning of a sentence, every absolute index shifts, forcing the model to relearn patterns it already understood.

To bridge this gap, researchers developed relative distance encoding (or relative position embeddings). By shifting from a global coordinate system to an entity-centric, relative coordinate system, models can preserve spatial awareness and capture the geometric layout of relationships within text.


The Limits of Absolute Coordinates

In tasks like relation extraction, event detection, and aspect-based sentiment analysis, the absolute index of a token provides little semantic value. As argued by Wang et al., absolute position embeddings frequently fail to capture the specific spatial relationships between words and target entities that are predictive of semantic links.

If a model only knows where words lie relative to the start of a document, it must infer the distance between entities implicitly through layers of transformations. This lack of explicit spatial context makes it difficult for encoders to distinguish between relevant local modifiers and background noise, particularly when sentences are long or syntactically complex.


Constructing an Entity-Centric Coordinate System

Relative distance encoding solves this by re-indexing the sentence from the perspective of the target entities. For every token in a sequence, the model calculates its structural distance (or offset) to the entities of interest.

For example, if a target entity resides at index 3 of a sentence, a context word at index 5 is assigned a relative distance of $+2$, while a word at index 1 receives a $-2$. These integer offsets are then mapped to a dedicated vocabulary of learnable relative position embeddings, which are combined directly with standard semantic word vectors.

Tokens:      The   company   acquired   the   startup .
Absolute:    [0]     [1]       [2]      [3]     [4]   [5]
---------------------------------------------------------
Relative to "company" (Index 1):
Distance:    -1      [0]       +1       +2      +3    +4
Relative to "startup" (Index 4):
Distance:    -4      -3        -2       -1      [0]   +1

This representational strategy was highly influential in adapting early deep learning architectures to NLP. Zeng et al. demonstrated that incorporating these relative distance encodings allowed Convolutional Neural Networks (CNNs)—which are inherently position-blind outside of their local filter windows—to maintain spatial awareness of subjects across an entire sentence. This explicit spatial tracking prevents vital structural information from being washed out during global pooling operations.

Similarly, in event detection tasks, the proximity of context words to a potential event trigger is a primary signal for classification. Nguyen et al. showed that removing relative position embeddings results in a severe drop in performance, proving that spatial context is just as vital as semantic content for identifying triggers. By providing dual relative distance features—tracking proximity to both subjects and objects simultaneously—the network can effectively map out the linguistic neighborhood of an interaction (Chen et al.; Dernoncourt et al.).


Preventing Feature Over-Aggregation

The utility of relative spatial context extends beyond linear sequence models into structural architectures like Graph Convolutional Networks (GCNs).

In aspect-based sentiment classification, GCNs are often deployed over dependency trees to aggregate features along syntactic paths. However, dependency trees can sometimes connect words that are syntactically related but physically distant in the actual text. Without a counterbalancing mechanism, the graph layers risk over-aggregating features from distant contexts, introducing noise into fine-grained sentiment boundaries.

To mitigate this, Xiao et al. integrated relative positional embeddings into GCN feature aggregation. By forcing the network to remain sensitive to the physical distance between tokens in the original sequence, the model maintains a healthy balance between syntactic relevance and physical proximity.

This spatial invariance also benefits generative tasks. For instance, in sequence-to-sequence dependency parsing, Li et al. configured the decoder to predict the relative distance and direction of a word's syntactic head rather than its absolute index. This design choice ensures that the learned parsing patterns remain invariant to sentence length or absolute position within a document.


Flipping Spatial Metadata into Output Labels

While relative distance embeddings began as an input feature to inform encoders, the underlying intuition has also been flipped to optimize output spaces.

In complex extraction tasks—such as Aspect Sentiment Triplet Extraction (ASTE), where a model must simultaneously extract an aspect, its corresponding opinion phrase, and the sentiment polarity—traditional pipelines require multiple cascading classification steps.

Xu et al. bypassed this complexity by baking relative spatial offsets directly into sequence tags. Their position-aware tagging scheme extends standard labels to include distance metadata (e.g., a tag indicating that a token is an aspect term and its associated opinion span begins exactly 3 tokens backward). By embedding the relational link as a relative distance within the tag itself, a standard sequence labeler can solve a multi-entity pairing task in a single pass.

Taking this geometric framing to its logical conclusion, Zhang et al. mapped word-pair relations onto a 2D grid, using convolutional layers to detect local spatial dependencies between entity boundaries. This line of work demonstrates that linguistic relationships can be treated as explicit spatial patterns, where relative distance serves as the foundational coordinate system.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 142 papers
  2. Version 1 · 2026-10-04 · 149 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_relative_position_embeddings_9c799b4e,
  title = {relative position embeddings},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relative-position-embeddings/rdr_9c799b4e3534478d97e1dce8dd7d5f8d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_relative_position_embeddings_9c799b4e_v2,
  title = {relative position embeddings},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relative-position-embeddings/rdr_9c799b4e3534478d97e1dce8dd7d5f8d/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake