Training a Large Language Model (LLM) to handle millions of tokens is not as simple as increasing a hyperparameter. The quadratic computational cost of self-attention and the instability of learning long-range dependencies from scratch make "long-context" training a significant engineering hurdle.
To overcome this, researchers have converged on a shared strategy: Staged Context Extension. Instead of training on massive sequences from day one, models follow a curriculum that progressively scales the context window. This approach balances computational efficiency with the need to capture complex, global dependencies.
The Efficiency of Starting Small
The intuition behind staged training is rooted in how models learn. In the early stages of pre-training, a model needs to master local syntax, grammar, and basic semantic relationships. These "local" features are present even in very short sequences.
By starting with short context windows—sometimes as small as 128 tokens—researchers like Hongye Jin and Xiaotian Han in GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Length demonstrate that models can process a much higher volume of tokens in the same wall-clock time. This "warm-up" phase builds a linguistic foundation efficiently before the model is tasked with the memory-intensive work of connecting ideas across thousands of pages.
This pattern is consistent across languages and architectures. For example, the Qwen2.5 Technical Report by An Yang describes a transition from 4,096 to 32,768 tokens to avoid destabilizing initial learning. Similarly, the development of The birth of Romanian BERT by Stefan Daniel Dumitrescu and IndoNLU by Bryan Wilie both utilized two-phase strategies to learn local context before introducing long-range dependencies.
Bridging the Gap with Intermediate Scaling
Moving directly from a 4K pre-training window to a 128K or 1M fine-tuning window often leads to performance degradation. To bridge this gap, many "recipes" introduce an intermediate adaptation phase.
In Scaling Granite Code Models to 128K Context, Matt Stallone and colleagues use a curriculum where the sequence length is doubled at each interval. This gradual stretching allows the model to adapt its positional embeddings and attention mechanisms incrementally.
Yao Zhao and Jason Phang found in Investigating Efficiently Extending Transformers for Long Input Summarization that this intermediate pre-training on moderately long sequences is more effective than direct fine-tuning. It acts as a "stretching exercise" for the model's internal representations, ensuring it can handle new attention patterns before facing the final target length.
Long Context as a Foundation for Reasoning
Recent evidence suggests that long-context ability is not just about "remembering" more; it is fundamentally tied to "thinking" better. In Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning, Van Yang argues that expanding the context window before reasoning-specific fine-tuning is critical. This ensures the model has the "cognitive space" to maintain coherence throughout the long, multi-step reasoning paths required for complex math and logic.
This synergy is exploited in models like Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models, which uses a staged context curriculum to stabilize the training of reasoning models. By gradually increasing the token budget for "thinking trajectories," the model learns to solve increasingly complex problems without the training instabilities that occur when processing massive sequences too early.
Specialized Post-Training and Annealing
The final stage of the extension pipeline often involves specialized Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL).
- Long-Context SFT: Liang Zhao in LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models highlights a critical long-context SFT phase that adapts the model's representational capacity to specific tasks like document retrieval or summarization.
- Context Annealing: The TeleChat2 Technical Report by Zihan Wang describes a curriculum-based annealing phase that increases sequence length while using high-quality curated data. This maintains performance on short-context tasks while enabling high accuracy in long-context retrieval.
- Evolutionary Search: For extreme extensions, Yiran Ding and Li Lyna Zhang introduced LongRoPE, which uses evolutionary search to identify non-uniformities in positional embeddings. This allows them to reach 2 million tokens with minimal fine-tuning on much shorter sequences (256k).
By treating context length as a curriculum rather than a fixed constraint, these staged approaches allow modern LLMs to scale from local word prediction to global document understanding and complex reasoning.