Staged Context Extension for Large Language Models

This direction is merged into another direction: temporal curriculum learning. Its last version stays readable and citable here.

Version 1 of 1265 papers326 conceptsfrozen 2026-10-04

Training a Large Language Model (LLM) to handle millions of tokens is not as simple as increasing a hyperparameter. The quadratic computational cost of self-attention and the instability of learning long-range dependencies from scratch make "long-context" training a significant engineering hurdle.

To overcome this, researchers have converged on a shared strategy: Staged Context Extension. Instead of training on massive sequences from day one, models follow a curriculum that progressively scales the context window. This approach balances computational efficiency with the need to capture complex, global dependencies.

The Efficiency of Starting Small

The intuition behind staged training is rooted in how models learn. In the early stages of pre-training, a model needs to master local syntax, grammar, and basic semantic relationships. These "local" features are present even in very short sequences.

By starting with short context windows—sometimes as small as 128 tokens—researchers like Hongye Jin and Xiaotian Han in GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Length demonstrate that models can process a much higher volume of tokens in the same wall-clock time. This "warm-up" phase builds a linguistic foundation efficiently before the model is tasked with the memory-intensive work of connecting ideas across thousands of pages.

This pattern is consistent across languages and architectures. For example, the Qwen2.5 Technical Report by An Yang describes a transition from 4,096 to 32,768 tokens to avoid destabilizing initial learning. Similarly, the development of The birth of Romanian BERT by Stefan Daniel Dumitrescu and IndoNLU by Bryan Wilie both utilized two-phase strategies to learn local context before introducing long-range dependencies.

Bridging the Gap with Intermediate Scaling

Moving directly from a 4K pre-training window to a 128K or 1M fine-tuning window often leads to performance degradation. To bridge this gap, many "recipes" introduce an intermediate adaptation phase.

In Scaling Granite Code Models to 128K Context, Matt Stallone and colleagues use a curriculum where the sequence length is doubled at each interval. This gradual stretching allows the model to adapt its positional embeddings and attention mechanisms incrementally.

Yao Zhao and Jason Phang found in Investigating Efficiently Extending Transformers for Long Input Summarization that this intermediate pre-training on moderately long sequences is more effective than direct fine-tuning. It acts as a "stretching exercise" for the model's internal representations, ensuring it can handle new attention patterns before facing the final target length.

Long Context as a Foundation for Reasoning

Recent evidence suggests that long-context ability is not just about "remembering" more; it is fundamentally tied to "thinking" better. In Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning, Van Yang argues that expanding the context window before reasoning-specific fine-tuning is critical. This ensures the model has the "cognitive space" to maintain coherence throughout the long, multi-step reasoning paths required for complex math and logic.

This synergy is exploited in models like Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models, which uses a staged context curriculum to stabilize the training of reasoning models. By gradually increasing the token budget for "thinking trajectories," the model learns to solve increasingly complex problems without the training instabilities that occur when processing massive sequences too early.

Specialized Post-Training and Annealing

The final stage of the extension pipeline often involves specialized Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL).

  • Long-Context SFT: Liang Zhao in LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models highlights a critical long-context SFT phase that adapts the model's representational capacity to specific tasks like document retrieval or summarization.
  • Context Annealing: The TeleChat2 Technical Report by Zihan Wang describes a curriculum-based annealing phase that increases sequence length while using high-quality curated data. This maintains performance on short-context tasks while enabling high accuracy in long-context retrieval.
  • Evolutionary Search: For extreme extensions, Yiran Ding and Li Lyna Zhang introduced LongRoPE, which uses evolutionary search to identify non-uniformities in positional embeddings. This allows them to reach 2 million tokens with minimal fine-tuning on much shorter sequences (256k).

By treating context length as a curriculum rather than a fixed constraint, these staged approaches allow modern LLMs to scale from local word prediction to global document understanding and complex reasoning.

Papers in this version

Showing 60 of 265 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_staged_context_extension_for_large_langu_69746152,
  title = {Staged Context Extension for Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/staged-context-extension-for-large-language-models/rdr_69746152221f47edb3dc0985fcf4dbd2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_staged_context_extension_for_large_langu_69746152_v1,
  title = {Staged Context Extension for Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/staged-context-extension-for-large-language-models/rdr_69746152221f47edb3dc0985fcf4dbd2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}