Disentangling the Expressivity of RoPE

Selim Jerad, Anej Svete, Jiaoda Li, Ryan Cotterell

TL;DR

This paper bridges the gap between theoretical and practical understandings of Rotary Position Embeddings (RoPE) in Transformers. By evaluating RoPE under a realistic finite-precision framework, the authors demonstrate that while periodic RoPE variants perfectly capture modular patterns, conventional RoPE is fundamentally non-periodic, acting instead as a bounded locality bias that can actually hinder long-range reasoning.

What Problem Does This Paper Address?

Rotary Position Embedding (RoPE) is the de facto standard positional encoding scheme in modern large language models, powering architectures like Llama 3, Qwen, and DeepSeek. Despite its massive success, there is a glaring contradiction in how researchers explain why it works:

  1. The Theoretical View: Expressivity studies praise RoPE for its mathematical periodicity, linking it to a model's ability to solve modular logic problems (e.g., tracking even/odd positions).
  2. The Practical View: Empirical and mechanistic studies show that RoPE's length generalization is often superficial, acting primarily as a local recency bias that helps models focus on nearby tokens but struggles with distant context.

This paper addresses this disconnect by formalizing both accounts within a uniform, finite-precision framework to determine what RoPE actually computes in practice.

Methods

The authors analyze soft-attention transformers under a fully uniform, fixed-precision framework ($\mathbb{F}$). They mathematically disentangle RoPE into two distinct regimes based on the choice of rotation angles:

  • Component-Periodic RoPE ($\text{RoPE}_\text{P}$): A variant where rotation angles are rational multiples of $2\pi$, causing the matrix rotations to eventually repeat.
  • Conventional Non-Periodic RoPE ($\text{RoPE}_\text{NP}$): The standard implementation used in practice (with bases like $\beta = 10^4$), which relies on irrational rotations that never repeat.

The authors map these regimes to formal logic frameworks, specifically fragments of Linear Temporal Logic (LTL). They prove that $\text{RoPE}_\text{P}$ is exactly equivalent to past temporal logic with modular predicates ($\text{LTL}[\text{P}, \text{MOD}]$). For conventional $\text{RoPE}_\text{NP}$, they prove it implements fixed-offset look-back operators (e.g., looking exactly $k$ steps back) up to a precision-dependent length bound called the certified range.

Datasets

The paper evaluates these theoretical insights using controlled synthetic formal languages designed to isolate specific linguistic and logical properties:

  • Modular Languages: $(aa)^*$ (strings of even length) and $(ab)^*$ (arbitrary repetitions of the substring $ab$).
  • Locality/Temporal Languages: $a\Sigma^*$ (strings starting with $a$), $\Sigma^*a$ (strings ending with $a$), and $\Sigma^*ab\Sigma^*$ (strings containing $ab$).

Models were trained on short sequences (lengths up to 40) and evaluated on longer sequences (lengths 41–500) to test true length extrapolation.

Key Insights and Ideas

  • The Periodicity Myth of Standard RoPE: While theoretical papers often assume RoPE is powerful due to periodicity, conventional RoPE never repeats because its phases are incommensurate with $2\pi$.
  • RoPE as a Bounded Locality Bias: Instead of providing an all-length characterization, conventional RoPE acts as a local look-back mechanism. It can perfectly identify a token exactly $k$ steps behind, but only up to a certain sequence length before finite-precision rounding causes numerical collisions.
  • The Long-Context Trade-off: Conventional RoPE improves performance on tasks requiring local recency but actively disrupts the position-invariant aggregation needed for long-distance context matching.
  • Attention Sinks as Phase Anchors: The authors reveal a novel theoretical reason for "attention sinks" (where models attend heavily to the initial BOS token): the BOS token acts as a static positional anchor, allowing the model to compute unary modular predicates by comparing moving query phases against a fixed key phase.

Key Results

  • Modular Tasks: $\text{RoPE}_\text{P}$ achieved 100% accuracy up to the maximum tested length of 500 on modular languages like $(aa)^*$ and $(ab)^*$. Conversely, conventional $\text{RoPE}_\text{NP}$ completely failed to length-generalize on these tasks across 24 orders of magnitude of the base parameter $\beta$.
Figure 2: Mean longest perfect length N∗for conventional non-periodic encodings on the modular languages. Markers are means over the run grid and whiskers are ±1 standard deviation. The two reference lines are best-run values for NoPE and RoPEP on these tasks. Neither non-periodic encoding generalizes perfectly at any tested base.
  • Locality Tasks: On the local recency task ($\Sigma^*a$), conventional $\text{RoPE}_\text{NP}$ outperformed a model with No Positional Encoding (NoPE). However, on tasks requiring global context aggregation ($a\Sigma^*$ and $\Sigma^*ab\Sigma^*$), NoPE significantly outperformed conventional RoPE, demonstrating that RoPE's local bias harms long-range reasoning.
Figure 1: The expressivity landscape of fully uniform finite-precision transformers. The red box shows our contributions: The connection of RoPEP and SiPEP transformers to LTL[P, MOD], which is incomparable with LTL[S] (capturing rightmost hard-attention NoPE transformers). Their intersection contains LTL[P], corresponding to soft-attention NoPE transformers.

Limitations and Future Work

  • Optimization vs. Expressivity: The periodic lower bound relies on a custom-engineered finite lookup table rather than proving that standard gradient descent can naturally discover this modular behavior from scratch.
  • Bounded Simulation: The locality analysis for non-periodic RoPE provides a bounded simulation rather than an asymptotic, all-length characterization.
  • Future Directions: The authors suggest exploring hybrid architectures (such as partial-RoPE or NoPE-RoPE hybrids) to combine the best of both worlds: strong local recency biases without sacrificing long-context global reasoning.

Summary updated

Abstract

Two accounts recur in explanations of the success of rotary position embeddings (RoPE). Expressivity studies associate periodic position information with modular predicates, whereas mechanistic and long-context studies emphasize positional anchors and local offsets. We formalize both accounts for fully uniform, finite-precision soft-attention transformers. We find that, if every rotary component is periodic, RoPE transformers recognize exactly the languages definable in past temporal logic with modular predicates. Conventional RoPE is different: The rotations it computes never repeat. This yields a precision-dependent bounded simulation of fixed-offset look-back operators, rather than an all-length modular characterization. Controlled experiments match this separation: Constructed periodic schedules length-generalize on modular languages, while conventional RoPE behaves more like a bounded locality bias and can impair tasks requiring position-invariant access to distant context. Altogether, our findings shed light on RoPE transformers, bringing theoretical expressivity characterizations closer to models used in practice.
BibTeX
@inproceedings{jerad2026disentangling,
  author = {Selim Jerad and Anej Svete and Jiaoda Li and Ryan Cotterell},
  title = {{Disentangling the Expressivity of RoPE}},
  booktitle = {COLM},
  year = {2026},
  url = {https://openreview.net/forum?id=HBOZmG9MR9},
}
Figure