Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models

Haoran Lian, Junmin Chen, Wei Huang, Yizhe Xiong, Wenping Hu, Guiguang Ding, Hui Chen, Jianwei Niu, Zijia Lin, Fuzheng Zhang, Di Zhang

Abstract

Recently, Large language models (LLMs) have revolutionized Natural Language Processing (NLP). Pretrained LLMs, due to limited training context size, struggle with handling long token sequences, limiting their performance on various downstream tasks. Current solutions toward long context modeling often employ multi-stage continual pertaining, which progressively increases the effective context length through several continual pretraining stages. However, those approaches require extensive manual tuning and human expertise. In this paper, we introduce a novel single-stage continual pretraining method, Head-Adaptive Rotary Position Encoding (HARPE), to equip LLMs with long context modeling capabilities while simplifying the training process. Our HARPE leverages different Rotary Position Encoding (RoPE) base frequency values across different attention heads and directly trains LLMs on the target context length. Extensive experiments on 4 language modeling benchmarks, including the latest RULER benchmark, demonstrate that HARPE excels in understanding and integrating long-context tasks with single-stage training, matching and even outperforming existing multi-stage methods. Our results highlight that HARPE successfully breaks the stage barrier for training LLMs with long context modeling capabilities.

Introduction

Large Language Models (LLMs) are typically pretrained on sequences of 2,048 to 4,096 tokens. This limitation severely hampers their utility in tasks requiring the processing of extensive documents, long-form codebases, or multi-turn conversations. To extend this context window, the prevailing industry standard involves multi-stage continual pretraining. This process involves incrementally increasing the context length (e.g., 8k $\rightarrow$ 16k $\rightarrow$ 32k $\rightarrow$ 128k) while adjusting the base frequency of the Rotary Position Encoding (RoPE).

However, multi-stage approaches are engineering-intensive, requiring meticulous manual tuning of hyperparameters and data scheduling at each transition. Attempting to jump directly from a short context (4k) to a very long one (128k) in a single stage typically leads to model instability or failure to converge.

Methodology: Head-Adaptive Rotary Position Encoding (HARPE)

We propose HARPE, a method that bypasses the need for multiple training stages by introducing "spatial" adaptivity within the model's architecture.

Head-Specific Base Frequencies

The core innovation of HARPE is the assignment of unique RoPE base frequencies ($\theta$) to different attention heads within the same layer. In traditional RoPE, all heads share a single base frequency. In HARPE, some heads are configured with low base frequencies to maintain short-range dependency modeling, while others are given high base frequencies to capture long-range dependencies. This allows the model to simulate the effects of multiple training stages simultaneously during a single training run.

Comparison between traditional multi-stage continual pretraining and the proposed single-stage HARPE method.

Selection Strategies

We investigated two primary strategies for distributing these base frequencies:

  1. Uniform Distribution: Linearly spacing base values between a minimum ($b_{min}$) and maximum ($b_{max}$).
  2. Peak-Valley Opposition: A more advanced strategy that ensures the periodic "valleys" (low attention regions) of one head's encoding are compensated for by the "peaks" (high attention regions) of another. This ensures continuous coverage across the entire long-context sequence.

Experimental Results

We evaluated HARPE using a 7B-parameter Llama-2 base model, training on 6 billion tokens from the SlimPajama dataset.

Performance on Upgraded Needle-in-a-Haystack

On the "Upgraded" NiaH benchmark, which involves multi-key and multi-value retrieval, HARPE achieved an average score of 86.82. This represents a 5.46% improvement over the traditional multi-stage Adjusted Base Frequency (ABF) approach, which averaged 81.36. Notably, HARPE maintains high performance even at the 128k token limit, where many other methods collapse.

Performance comparison on the upgraded Needle-in-a-Haystack (NiaH) benchmark across various context lengths.

RULER and Perplexity

On the challenging RULER benchmark, HARPE achieved an average score of 70.40, outperforming established models like LWM (67.67) and LongLoRA (60.63), ranking 1st among the 7B-parameter models compared. Furthermore, perplexity (PPL) evaluations on long-document datasets like Proof-pile and GovReport showed that HARPE maintains superior next-token prediction capabilities compared to both single-stage and multi-stage baselines.

Conclusion

HARPE successfully demonstrates that LLMs can acquire long-context capabilities in a single training stage without the complexity of multi-stage pipelines. By leveraging head-adaptive positional encodings, HARPE achieves state-of-the-art results on challenging benchmarks like RULER while maintaining performance on short-context reasoning tasks. This approach significantly simplifies the process of extending LLM context windows, paving the way for more efficient development of long-context models.

Summary updated

Abstract

Recently, Large language models (LLMs) have revolutionized Natural Language Processing (NLP). Pretrained LLMs, due to limited training context size, struggle with handling long token sequences, limiting their performance on various downstream tasks. Current solutions toward long context modeling often employ multi-stage continual pertaining, which progressively increases the effective context length through several continual pretraining stages. However, those approaches require extensive manual tuning and human expertise. In this paper, we introduce a novel single-stage continual pretraining method, Head-Adaptive Rotary Position Encoding (HARPE), to equip LLMs with long context modeling capabilities while simplifying the training process. Our HARPE leverages different Rotary Position Encoding (RoPE) base frequency values across different attention heads and directly trains LLMs on the target context length. Extensive experiments on 4 language modeling benchmarks, including the latest RULER benchmark, demonstrate that HARPE excels in understanding and integrating long-context tasks with single-stage training, matching and even outperforming existing multi-stage methods. Our results highlight that HARPE successfully breaks the stage barrier for training LLMs with long context modeling capabilities.
BibTeX
@inproceedings{lian2025breaking,
  author = {Haoran Lian and Junmin Chen and Wei Huang and Yizhe Xiong and Wenping Hu and Guiguang Ding and Hui Chen and Jianwei Niu and Zijia Lin and Fuzheng Zhang and Di Zhang},
  title = {{Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=Qix4DhK3t4},
}
Figure