Round and Round We Go! What makes Rotary Positional Encodings useful?

Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, Petar Veličković

TL;DR

This paper challenges the common assumption that Rotary Positional Encodings (RoPE) are effective because they naturally cause attention to decay with distance. By analyzing a trained Gemma 7B model, the authors reveal that LLMs actually use RoPE's highest frequencies to build sharp positional attention heads and its lowest frequencies for semantic processing. Based on these insights, they introduce p-RoPE, a modified encoding that improves model performance by truncating the lowest, least robust frequencies.

What Problem Does This Paper Address?

Transformers require a way to understand the order of tokens in a sequence. Rotary Positional Encodings (RoPE) have become the gold standard for this in modern Large Language Models (LLMs) like Llama 3 and Gemma. RoPE works by splitting query and key vectors into 2D chunks and rotating them at different frequencies based on their position in the sequence.

When RoPE was introduced, its creators argued that its primary benefit was "long-term decay"—the mathematical property where attention weights naturally decrease as the distance between tokens grows. However, this paper points out a major catch: that mathematical proof assumes queries and keys are constant vectors. In reality, LLM activations look much more like random Gaussian vectors. The authors set out to answer a fundamental question: If RoPE doesn't actually cause distance-based decay in practice, why is it so incredibly useful?

Methods

The authors take a mechanistic interpretability approach to "reverse-engineer" how a pretrained Gemma 7B model utilizes RoPE at a mechanical level. They analyze the 2-norm (magnitude) of query and key vectors across different rotation frequencies to see which frequencies the model prioritizes.

They complement this empirical analysis with mathematical proofs demonstrating:

  1. RoPE's capacity to construct perfectly sharp positional attention patterns.
  2. The inherent instability of RoPE's low-frequency channels over very long contexts.

Finally, they propose a novel variant called p-RoPE (partial-RoPE). Instead of rotating every dimension, p-RoPE truncates a percentage ($1-p$) of the lowest frequencies, leaving those channels completely unrotated (behaving like No Positional Encoding, or NoPE) to serve as stable, distance-agnostic semantic channels.

Datasets

  • Mechanistic Analysis: Conducted using a pretrained Gemma 7B model evaluated on text sequences from Shakespeare quotes.
  • p-RoPE Evaluation: To test their new encoding, the authors trained Gemma 2B models from scratch for 10,000 steps on two major datasets:
  • English Wikipedia (Wiki): A massive corpus of clean, encyclopedic text.
  • FlanV2: A popular dataset for instruction tuning covering a wide variety of tasks.

Key Insights and Ideas

  • The Decay Myth Busted: The authors show that when RoPE is applied to realistic, Gaussian-distributed queries and keys, the expected attention activation is exactly 0, completely independent of distance. True decay only happens on unrealistic, constant "all-ones" vectors.
Figure 2: RoPE applied to either (a) constant ‘all-ones’ queries and keys or (b) queries and keys with entries sampled IID from a Gaussian. The decay of the activations is present when the queries and keys are constant all-ones vectors, but not when they are Gaussian random vectors.
  • High Frequencies Build Position Circuits: The model cleverly exploits RoPE's highest frequencies (which rotate rapidly, up to 1 radian per token) to build sharp, purely positional attention heads—like diagonal "attend-to-self" or "attend-to-previous-token" heads. The authors prove mathematically that standard attention without positional encodings (NoPE) is completely incapable of forming these sharp patterns in isolation.
Figure 1: Depiction of our construction which allows Transformers to obtain positional attention heads using RoPE – zooming in on a single RoPE frequency for clarity. On the left we depict key and query vectors for each position i, where keys are all identical, and queries are just a rotated version of the key, in a way that matches one of RoPE’s highest frequencies. The center depicts how keys get rotated by RoPE, making the key at i´1 perfectly align with the query. Due to the high frequency of the rotation, all other keys will lead to a smaller attention weight. On the right we show the resulting attention weights, resulting in this case in an off-diagonal positional attention. See Section 5 for more details.
  • Low Frequencies Act as Semantic Channels: Across almost all layers, Gemma heavily favors the lowest frequencies of RoPE, dedicating the highest vector norms to them. Because these frequencies rotate incredibly slowly, they remain relatively invariant over short distances, allowing the model to use them to match token meanings (semantics).
  • The Long-Context Vulnerability: Because these low frequencies do still rotate slightly, they eventually misalign over very long context windows. This explains why modern models like Llama 3 have to artificially scale up the "base wavelength" parameter ($\theta$) to 500,000 for long-context windows: it slows down the rotation of these semantic channels even further to keep them stable.

Key Results

The authors validated their insights by training Gemma 2B models from scratch and comparing standard RoPE, NoPE, and their proposed p-RoPE.

  • p-RoPE Outperforms Standard RoPE: Truncating the lowest frequencies yielded the best results. Keeping 75% of the frequencies (0.75-RoPE) achieved a validation perplexity of 4.4414 on Wiki and 6.4422 on FlanV2.
  • Comparison to Baselines: This outperformed standard RoPE ($\theta = 10k$, which scored 4.4627 / 6.4429) and even beat the popular long-context trick of simply increasing the base wavelength to 500k ($\theta = 500k$, which scored 4.4485 / 6.4593).
  • NoPE Lags Behind: Models trained completely without positional encodings (NoPE) performed significantly worse (4.8594 / 6.6429), confirming that the positional circuits built by high frequencies are indeed critical.

Limitations and Future Work

The primary limitation acknowledged by the authors is compute scale. Due to resource constraints, they evaluated p-RoPE on a standard 8k token context window using a 2B parameter model. While their results strongly suggest that removing low-frequency rotations creates more robust semantic channels, the true test of this method will be scaling it to massive context lengths (e.g., 32k to 128k+ tokens) where low-frequency misalignment is most destructive. Additionally, evaluating downstream task accuracy rather than just validation perplexity remains an open item for future work.

Summary updated

Abstract

Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust "positional" attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments to verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling LLMs to large sizes and context lengths.
BibTeX
@inproceedings{barbero2025round,
  author = {Federico Barbero and Alex Vitvitskyi and Christos Perivolaropoulos and Razvan Pascanu and Petar Veličković},
  title = {{Round and Round We Go! What makes Rotary Positional Encodings useful?}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=GtvuNrk58a},
}
Figure