Abstract
Recent trends in Large Language Model (LLM) pretraining emphasize expanding context windows to handle increasingly long sequences. However, training on long contexts from the outset often compromises model performance and computational efficiency. We present SkyLadder, a novel pretraining strategy that employs a dynamic context window schedule. By starting with a minimal window and linearly scaling to the target length, SkyLadder achieves up to 3.7% higher benchmark accuracy and 22% faster training speeds compared to fixed-length baselines. Our findings suggest that "starting small" acts as a critical stabilizer for optimization and leads to more effective attention patterns.
Introduction
The AI industry is currently engaged in an "arms race" to expand LLM context windows, with modern models frequently handling hundreds of thousands of tokens. The prevailing assumption is that training on long sequences from the start is necessary to capture long-range dependencies.
However, empirical evidence suggests a significant trade-off: when holding the total training token budget constant, models trained on shorter context windows consistently outperform their long-context counterparts on standard benchmarks. Long-context training is computationally expensive, prone to optimization instabilities like loss spikes, and often results in "diluted" attention patterns. SkyLadder addresses this by bridging the gap between the efficiency of short-context training and the capabilities of long-context models.
Methodology
SkyLadder introduces a context window scheduling strategy where the model's window size $w(t)$ grows over training time $t$.
The schedule begins with a very short window (e.g., $w_s = 8$ tokens) and expands linearly until reaching the target length (e.g., $w_e = 32,768$ tokens). The authors recommend allocating approximately 60% of the total training tokens to this expansion phase.
To maintain implementation efficiency and avoid domain biases from data repacking, SkyLadder utilizes local causal masking. This technique applies a mask to long sequences, restricting tokens to attend only to a local "mini-window" that expands as training progresses. This approach significantly reduces the $O(n^2)$ complexity of the attention mechanism during early training stages, leading to substantial wall-clock time savings.
Key Insights
- Stability Warmups: Starting with small windows acts as a natural stabilizer. Models trained on 16K contexts from the start showed higher loss volatility and exploding attention logits, whereas SkyLadder models remained stable.
- Attention Concentration: SkyLadder models exhibit lower attention entropy, meaning they learn to focus more intensely on relevant information rather than spreading attention thinly across a massive window.
- Delaying Attention Sinks: The model delays the emergence of "attention sinks" (dumping attention on the first token), which leads to more effective feature learning.
Results
SkyLadder demonstrates superior performance and efficiency across scales:
- Performance Boost: A 1B-parameter SkyLadder model improved average benchmark scores by 3.7% over a random-packing baseline.
- Speedup: Training was up to 22% faster for 32K context models and 13% faster for 8K models compared to constant-length baselines.
- Scaling: Benefits persisted and even increased as models scaled from 120M to 3B parameters.
- Long-Context Parity: Despite the short-context start, SkyLadder models matched or exceeded native long-context models on tasks like MDQA and RULER.
Conclusion
SkyLadder proves that scheduling the context window is a more effective approach than fixed-length pretraining. While validated up to 3B parameters and 32K tokens, future research is needed to test these dynamics at frontier scales (70B+ parameters) and extreme context lengths (1M+ tokens). Additionally, formal theoretical analysis of the optimal schedule shape remains an open area for investigation.
Summary updated