TL;DR
Standard Rotary Position Embeddings (RoPE) make attention scores position-relative but leave the value pathway completely position-blind. This paper introduces RoVE (Rotary Value Embeddings), a parameter-free modification that rotates value vectors alongside keys and queries, transforming standard attention into an "attentive convolution." This simple change yields consistent empirical gains in long-context retrieval, out-of-distribution perplexity, and few-shot in-context learning.
What Problem Does This Paper Address?
In modern Large Language Models, RoPE has become the de facto standard for encoding positional information. By rotating query ($Q$) and key ($K$) vectors based on their absolute positions, RoPE ensures that the resulting attention score depends only on the relative distance (offset) between tokens.
However, the authors point out a fundamental asymmetry: RoPE only updates the QK circuit (which decides where to look), leaving the OV (Output-Value) circuit (which decides what message to send) entirely position-blind. Once an attention weight is computed, the value vector ($V$) fetched from a token is identical regardless of whether that token is right next to the query or thousands of tokens away. This lacks the shift-equivariance inherent to classical convolution kernels, where the transformation applied to a feature depends on its spatial or temporal offset.
Methods
The authors propose RoVE (Rotary Value Embeddings), a parameter-free extension to RoPE. Instead of leaving the value pathway untouched, RoVE rotates the value vector at position $j$ by the rotation matrix $R_j$ before aggregation, and then applies the inverse rotation $R_i^{-1}$ at the query position $i$ after aggregation:
$$\tilde{y}_i = R_i^{-1} \sum_{j \in \mathcal{N}(i)} A(X)_{ij} R_j W_V x_j = \sum_{j \in \mathcal{N}(i)} A(X)_{ij} R_{j-i} W_V x_j$$
By utilizing the properties of rotation matrices ($R_i^{-1} R_j = R_{j-i}$), the value transformation becomes explicitly dependent on the relative offset $\delta = j - i$. This effectively replaces the constant value projection matrix $W_V$ with an offset-indexed, block-Toeplitz kernel family $\psi_\delta = R_\delta W_V$.
Computational Efficiency
Because rotations act independently on individual tokens and channel pairs, RoVE adds a negligible computational overhead of $O(nd)$ complexity. Crucially, because the rotations are applied before and after the core attention aggregation, RoVE remains fully compatible with hardware-accelerated kernels like FlashAttention.
Datasets
The authors evaluated RoVE by training 124M and 354M parameter GPT-2 style models from scratch on:
- FineWebEdu-10B: Used for pre-training (1 epoch, 1024 context length) and out-of-distribution (OOD) perplexity evaluation.
- DCLM-Core: A diverse suite of few-shot tasks used to evaluate in-context learning (ICL) accuracy within the training context.
- RULER: A synthetic long-context benchmark consisting of four tasks: Common Word Extraction (CWE), multi-key Needle-in-a-Haystack (NIAH), Question Answering (QA), and Variable Tracking (VT).
Key Insights and Ideas
- The Attentive Convolution Perspective: RoVE bridges the gap between self-attention and convolutions. It mathematically unifies standard RoPE attention with the framework of attentive convolutions, where content-dependent scalar weights gate a structurally constrained, relative-position-dependent kernel.
- Unifying Independent Discoveries: "RoPE-on-values" mechanisms have been independently derived in niche applications across computer vision, robotics, and even in DeepSeek-V4's compressed shared-KV architecture (where it was used as a corrective measure for positional leakage). This paper provides the first rigorous, structural account of the mechanism within the transformer circuits framework.
- Complementarity with YaRN: RoVE addresses a completely different aspect of long-context generalization than frequency-interpolation methods like YaRN. While YaRN rescales frequencies to handle out-of-distribution token distances in the QK circuit, RoVE ensures the OV circuit structurally matches this relative-position awareness. When combined, they achieve state-of-the-art long-context performance.
Key Results
The authors compared standard RoPE against RoVE across multiple scales. For the 354M parameter model:
- In-Context Learning: RoVE improved the DCLM-Core accuracy score from 0.1664 to 0.1856 within the standard 1024 training window, showing that the relative-position bias helps even on short-context tasks.
- Context Window Extrapolation: When evaluating out-of-distribution perplexity at a 16k context length (16× training length) without fine-tuning:
- Standard RoPE exploded to a perplexity of 1630.72.
- RoVE naturally mitigated this, holding a perplexity of 583.84.
- When paired with YaRN interpolation, RoVE + YaRN achieved a perplexity of 124.82, vastly outperforming RoPE + YaRN (270.98).
- Long-Context Retrieval (RULER): On tasks requiring tracking and recombining information across long sequences (at 8k tokens), RoVE + YaRN reduced the mean negative log-likelihood (NLL) from 6.62 (RoPE + YaRN) to 4.33. The most pronounced improvements occurred in complex tasks like Variable Tracking and multi-key Needle-in-a-Haystack.
Limitations and Future Work
While RoVE demonstrates clear empirical benefits, the authors note that the exact mathematical mechanism driving the massive reduction in OOD perplexity is not yet fully understood. Their current working hypothesis is that sharing the same rotation family across both the QK and OV pathways prevents the circuits from "drifting apart" at unseen offsets.
Future work will focus on analyzing errors across specific RoPE frequency bands to verify this hypothesis. Additionally, exploring the combination of RoVE with semantics-aware QK circuit modifications (such as Polar Position Embeddings / PoPE) represents a promising avenue for completely disentangling "what" from "where" in transformers.
Summary updated