Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE

Mohammad Aflah Khan, Krishna P. Gummadi, Manish Gupta, Abhilasha Ravichander

Abstract

Rotary Positional Embedding (RoPE) is a common choice in transformer architectures for encoding relative positional information. Although earlier work has examined omitting RoPE in specific layers, the effect of varying the fraction of hidden dimensions that receive rotary transformations remains largely unexplored. This design choice can yield substantial memory savings, which becomes especially significant at long context lengths. We find up to 10x memory savings over the standard RoPE cache, while achieving comparable final loss. In this work, we present a systematic study examining the impact of partial RoPE on training dynamics and convergence across architectures and datasets. Our findings uncover several notable patterns: (1) applying RoPE to only a small fraction of dimensions (around 10%) achieves convergence comparable to using full RoPE; (2) these trends hold consistently across model size, sequence lengths and datasets of varying quality and architectures; and (3) some models trained with NoPE (No Positional Encoding) showcase unstable learning trajectories, which can be alleviated through minimal RoPE application or QK-Norm.

Introduction

In Large Language Models (LLMs), Rotary Positional Embedding (RoPE) has become the industry standard for encoding token order. While models like LLaMA rotate 100% of their hidden dimensions, others like GPT-NeoX or Phi-2 use "partial RoPE," rotating only 25% or 40%. This design choice has massive implications for efficiency. RoPE requires a "sine/cosine cache" that scales linearly with sequence length. As context windows push toward millions of tokens, this cache consumes substantial VRAM, particularly on memory-constrained edge devices. This study identifies the "sweet spot" for partial RoPE that balances memory efficiency with model performance.

Methodology

We conducted a systematic empirical study by pre-training several models from scratch, comparing RoPE application fractions: 0% (NoPE), ~1%, 10%, 25%, 50%, 75%, and 100% (Full RoPE).

To ensure robustness, we tested:

  • Architectures: Sequential attention (Llama-style) and Parallel attention (Pythia-style).
  • Scales: 1B and 8B parameter models.
  • Sequence Lengths: From 1,024 to 8,192 tokens.
  • Datasets: FineWeb (100B tokens) and FineWeb-Edu.
  • Stability Interventions: Investigated QK-Norm (Query-Key Normalization) for stabilizing models without positional encodings.

Evaluation was performed using the EleutherAI LM Evaluation Harness across benchmarks like ARC, PIQA, SciQ, WinoGrande, and LAMBADA.

Key Results

The 10% Performance Plateau

A striking finding is that applying RoPE to just 10% of dimensions matches the convergence and final loss of 100% RoPE. Diminishing returns occur beyond this 10% threshold. Models with 10% to 100% RoPE form a "tight cluster" in training loss, performing essentially identically across most downstream tasks.

Training loss convergence and instability at 8192 sequence length.

Memory Efficiency Gains

Moving from 100% to 10% RoPE yields a 10x reduction in the VRAM required for the RoPE cache. This is critical for deploying ultra-long-context models on consumer hardware, where VRAM is often the primary bottleneck.

VRAM savings of partial RoPE across sequence lengths.

Training Stability and the "NoPE" Danger Zone

Training without positional encoding (NoPE) or with fractions under 4% leads to significantly higher final loss and occasional catastrophic "loss spikes." While QK-Norm can prevent these spikes, it does not close the performance gap. Partial RoPE (at 10%+) remains the superior strategy for achieving both stability and low loss.

Conclusion

Our results offer practical guidance for model designers: applying RoPE to approximately 10% of hidden dimensions provides the full benefits of positional encoding while maximizing memory efficiency. This trend holds across model scales and sequence lengths, suggesting that partial RoPE is a viable and highly effective strategy for future large-scale transformer architectures.

Summary updated

Abstract

Rotary Positional Embedding (RoPE) is a common choice in transformer architectures for encoding relative positional information. Although earlier work has examined omitting RoPE in specific layers, the effect of varying the fraction of hidden dimensions that receive rotary transformations remains largely unexplored. This design choice can yield substantial memory savings, which becomes especially significant at long context lengths. We find up to 10x memory savings over the standard RoPE cache, while achieving comparable final loss. In this work, we present a systematic study examining the impact of partial RoPE on training dynamics and convergence across architectures and datasets. Our findings uncover several notable patterns: (1) applying RoPE to only a small fraction of dimensions (around 10%) achieves convergence comparable to using full RoPE; (2) these trends hold consistently across model size, sequence lengths and datasets of varying quality and architectures, with higher-quality data resulting in lower overall loss and similar benchmark performance; and (3) some models trained with NoPE (No Positional Encoding) showcase unstable learning trajectories, which can be alleviated through minimal RoPE application or QK-Norm which converges to a higher loss. Together, these results offer practical guidance for model designers aiming to balance efficiency and training stability, while emphasizing the previously overlooked importance of partial RoPE.
BibTeX
@misc{khan2026fractional,
  author = {Mohammad Aflah Khan and Krishna P. Gummadi and Manish Gupta and Abhilasha Ravichander},
  title = {{Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE}},
  year = {2026},
  eprint = {2603.11611},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.11611},
}
Figure