Abstract
Rotary Positional Embedding (RoPE) is a common choice in transformer architectures for encoding relative positional information. Although earlier work has examined omitting RoPE in specific layers, the effect of varying the fraction of hidden dimensions that receive rotary transformations remains largely unexplored. This design choice can yield substantial memory savings, which becomes especially significant at long context lengths. We find up to 10x memory savings over the standard RoPE cache, while achieving comparable final loss. In this work, we present a systematic study examining the impact of partial RoPE on training dynamics and convergence across architectures and datasets. Our findings uncover several notable patterns: (1) applying RoPE to only a small fraction of dimensions (around 10%) achieves convergence comparable to using full RoPE; (2) these trends hold consistently across model size, sequence lengths and datasets of varying quality and architectures; and (3) some models trained with NoPE (No Positional Encoding) showcase unstable learning trajectories, which can be alleviated through minimal RoPE application or QK-Norm.
Introduction
In Large Language Models (LLMs), Rotary Positional Embedding (RoPE) has become the industry standard for encoding token order. While models like LLaMA rotate 100% of their hidden dimensions, others like GPT-NeoX or Phi-2 use "partial RoPE," rotating only 25% or 40%. This design choice has massive implications for efficiency. RoPE requires a "sine/cosine cache" that scales linearly with sequence length. As context windows push toward millions of tokens, this cache consumes substantial VRAM, particularly on memory-constrained edge devices. This study identifies the "sweet spot" for partial RoPE that balances memory efficiency with model performance.
Methodology
We conducted a systematic empirical study by pre-training several models from scratch, comparing RoPE application fractions: 0% (NoPE), ~1%, 10%, 25%, 50%, 75%, and 100% (Full RoPE).
To ensure robustness, we tested:
- Architectures: Sequential attention (Llama-style) and Parallel attention (Pythia-style).
- Scales: 1B and 8B parameter models.
- Sequence Lengths: From 1,024 to 8,192 tokens.
- Datasets: FineWeb (100B tokens) and FineWeb-Edu.
- Stability Interventions: Investigated QK-Norm (Query-Key Normalization) for stabilizing models without positional encodings.
Evaluation was performed using the EleutherAI LM Evaluation Harness across benchmarks like ARC, PIQA, SciQ, WinoGrande, and LAMBADA.
Key Results
The 10% Performance Plateau
A striking finding is that applying RoPE to just 10% of dimensions matches the convergence and final loss of 100% RoPE. Diminishing returns occur beyond this 10% threshold. Models with 10% to 100% RoPE form a "tight cluster" in training loss, performing essentially identically across most downstream tasks.
Memory Efficiency Gains
Moving from 100% to 10% RoPE yields a 10x reduction in the VRAM required for the RoPE cache. This is critical for deploying ultra-long-context models on consumer hardware, where VRAM is often the primary bottleneck.
Training Stability and the "NoPE" Danger Zone
Training without positional encoding (NoPE) or with fractions under 4% leads to significantly higher final loss and occasional catastrophic "loss spikes." While QK-Norm can prevent these spikes, it does not close the performance gap. Partial RoPE (at 10%+) remains the superior strategy for achieving both stability and low loss.
Conclusion
Our results offer practical guidance for model designers: applying RoPE to approximately 10% of hidden dimensions provides the full benefits of positional encoding while maximizing memory efficiency. This trend holds across model scales and sequence lengths, suggesting that partial RoPE is a viable and highly effective strategy for future large-scale transformer architectures.
Summary updated