Deconstructing Positional Information: From Attention Logits to Training Biases

Zihan Gu, Ruoyu Chen, Han Zhang, Hua Zhang, Yue Hu

TL;DR

Researchers have deconstructed how Transformers process positional information, revealing a fundamental divide between additive (e.g., ALiBi) and multiplicative (e.g., RoPE) encodings. While RoPE’s multiplicative nature makes it superior at coupling content with position, it suffers from a "single-head deposit pattern"—a training bias where nearly all positional logic in early layers collapses into one hyper-specialized attention head.

The Problem: How Do Transformers Know "Where" Words Are?

Transformers are naturally "bag-of-words" models; they don't inherently understand the order of tokens unless explicitly instructed. We bridge this gap using Positional Encodings (PE). While many flavors of PE exist—such as Absolute embeddings, T5 biases, ALiBi, and the now-dominant Rotary Positional Embedding (RoPE)—the "mechanistic" differences in how they handle the interplay between what a word is (content) and where it is (position) remain poorly understood.

Current benchmarks often rely on aggregate metrics like perplexity, which are too noisy to reveal how these signals are processed internally. To address this, the authors provide a unified mathematical framework to analyze PE, explaining why certain schemes like RoPE exhibit strange, hyper-localized behaviors during training.

Methods & Analytical Framework

The authors introduce a unified analytical framework based on Toeplitz matrices. Through this lens, any translation-invariant positional signal (where the relationship depends only on the relative distance between tokens) can be represented as a Toeplitz structure within the attention logit matrix.

Overview of the two modes of positional encoding (Additive vs. Multiplicative) and their resulting effects on generalization and attention head behavior.

Using this framework, they categorize PE into two distinct modes:

  1. Additive Mechanisms (Absolute PE, T5, ALiBi): These add a static bias or position vector directly to the logits. Content and position mostly interact through cross-terms, keeping the signals somewhat isolated.
  2. Multiplicative Mechanisms (RoPE): These use a Hadamard product with a Toeplitz kernel. This allows the model to "modulate" content interactions based on position, creating a much tighter, dynamic coupling between the two.

To observe these mechanisms in action, the researchers employed causal head-wise ablation: training a model and systematically "zeroing out" specific attention heads to isolate where positional information lives.

Targeted Synthetic Datasets

Because natural language is too complex to cleanly isolate these variables, the authors designed two targeted synthetic tasks:

  • Task 1 (Position-Sensitive - Relative Distance Classification): The model must identify two "trigger" words and predict the distance between them. This requires strong content-position coupling.
  • Task 2 (Position-Agnostic - Trigger Word Counting): The model counts occurrences of specific words regardless of their location. This tests if a PE scheme is "too loud" and interferes with simple semantic tasks.

Key Insights: The "Deposit Pattern"

  • The "Single-Head Deposit Pattern": This is the paper's most striking discovery. In RoPE-equipped models, positional reasoning in the shallow layers is not evenly distributed. Instead, the model "deposits" almost all positional logic into a single attention head. If you ablate this specific head in Layer 1, the model’s accuracy on position tasks collapses by ~60%, while ablating any other head has near-zero effect.
  • RoPE’s Expressive Power is a Double-Edged Sword: RoPE allows for a much larger solution space than additive methods, making it vastly superior for tasks requiring precise relative positioning. However, this creates a "winner-takes-all" dynamic during training that forces the deposit pattern.
  • Storage of Information: Additive PEs store positional information in the activations (explicit directions in the vector space). RoPE, conversely, rapidly suppresses these activation-level directions and instead embeds the positional structure into the frequency-modulated parameters of the query/key projections.
  • Hybrid Architectures as a Solution: The authors note that architectures like DeepSeek’s Multi-Latent Attention (MLA)—which uses a hybrid of NoPE and RoPE pathways—successfully mitigate the deposit pattern, distributing positional responsibility more robustly across the model.

Key Results

Training and testing accuracy on the position-sensitive task, highlighting RoPE's significant performance advantage.
  • Performance Gap: On the position-sensitive task, RoPE achieved an impressive 92.6% accuracy, while models with no PE (NoPE) or standard Relative PE failed completely (hovering around ~3.5%).
  • Inductive Bias: On the position-agnostic counting task, strong additive biases like ALiBi were actually detrimental. RoPE’s adaptive nature, however, allowed it to effectively "ignore" position when unnecessary and perform well.
  • Gradient Amplification: Theoretical analysis proved that RoPE’s multiplicative structure causes an exponential amplification of the gradient signal for the "dominant" head during backpropagation. This explains why the single-head deposit pattern is an intrinsic mathematical property of the architecture, not a random training fluke.

Limitations and Future Work

The authors acknowledge that their findings are currently based on synthetic tasks. While these tasks are excellent for isolating variables, the "deposit pattern" needs to be verified in massive, real-world LLMs trained on natural language corpora.

Additionally, they suggest that the "massive value" phenomenon (where a few rows in weight matrices develop huge norms) might be a direct byproduct of this frequency bottleneck in RoPE. Future work will likely explore whether breaking this single-head bottleneck can improve a model's ability to handle complex algorithmic tasks, such as Dyck language recognition or long-context reasoning.

Summary updated

Abstract

Positional encodings, a mechanism for incorporating sequential information into the Transformer model, are central to contemporary research on neural architectures. Previous work has largely focused on understanding their function through the principle of distance attenuation, where proximity dictates influence. However, the interaction between positional and semantic information remains insufficiently explored, and the complexity of mainstream corpora hinders systematic, comparative studies of these methods. This paper addresses these challenges through a deconstruction of the attention-logit computation and a structured analysis of all mainstream positional encodings. A key focus is placed on Rotary Positional Embedding (RoPE), whose product-based structure uniquely facilitates a direct interaction between position and content. To probe this characteristic, we designed a novel synthetic task that explicitly demands a strong synthesis of positional and semantic information. As theoretically predicted, RoPE demonstrates a significant performance advantage over other encodings on this specialized task. Concurrently, this targeted evaluation uncovers an implicit training issue: a hidden bias manifesting as a distinct information aggregation phenomenon in the model's shallow layers, which we term the "single-head deposit pattern." Through subsequent ablation studies, we analyze this pattern and identify a method for its mitigation. These findings highlight the need for a deeper investigation into the training dynamics of positional encodings to bridge the gap between their theoretical design and practical implementation.
BibTeX
@inproceedings{gu2026deconstructing,
  author = {Zihan Gu and Ruoyu Chen and Han Zhang and Hua Zhang and Yue Hu},
  title = {{Deconstructing Positional Information: From Attention Logits to Training Biases}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=D0u0glT060},
}
Figure