Progressive Sequence Length Training

674 concepts434 papers7 proposals

Training sequence models on long contexts is notoriously difficult. For Transformers, the computational cost of self-attention scales quadratically ($O(N^2)$) with the sequence length $N$. For recurrent architectures, long sequences compound optimization challenges, making it difficult to backpropagate gradients through extended temporal horizons without encountering exploding or vanishing gradients.

Faced with these bottlenecks, a natural question arises: Why force a model to parse complex, long-range dependencies before it has even mastered basic local structures?

Progressive sequence length training—often framed as a length-based curriculum—addresses this by starting the training process on short sequences and incrementally increasing the context window over time. This approach provides a powerful framework for accelerating training, stabilizing optimization, and unlocking massive context windows.


Why It Works: The Mechanics of Length Curriculums

The success of progressive sequence length training relies on three core insights: token throughput maximization, gradient stabilization, and hierarchical feature learning.

1. Maximizing Token Throughput

In the early stages of training, a model's parameters are highly unoptimized, and it must process vast amounts of data to learn fundamental statistical patterns (such as word co-occurrences or local pixel correlations). Processing short sequences requires significantly less memory and computational overhead per batch.

As shown by Jin et al. in their work on GrowLength, training on shorter sequence lengths early on drastically increases token throughput. By utilizing the lower computational cost of short sequences during the initial phases, models can ingest a much higher volume of data within the same wall-clock time compared to fixed-length training, leading to a faster initial drop in loss.

2. Natural Gradient Warm-up

Exposing an uninitialized or partially trained model to massive context windows can cause severe optimization instability, frequently manifesting as sudden gradient spikes. A length-based curriculum acts as a natural warm-up mechanism.

By gradually increasing the sequence length, as explored by Pouransari et al. in their dataset decomposition framework, the training pipeline avoids sudden shifts in positional complexity. This monotonic growth allows the model's internal representations to adapt smoothly, preventing gradient spikes and enabling the use of higher learning rates during the critical early phases of optimization.

3. Hierarchical Feature Learning

Sequence understanding is inherently hierarchical. To understand a full page of text, a model must first understand phrases and sentences; to separate mixed audio over minutes, it must first capture local spectral features.

A multi-phase training strategy aligns perfectly with this hierarchy. Early phases focus on shorter sequence lengths to efficiently capture local context and short-range dependencies. Once these foundational representations are stable, the final phases introduce longer sequences to teach the model how to bridge long-range dependencies.

This phased approach has been a staple in building robust language models across various languages, including early iterations of WikiBERT (Pyysalo et al.), GigaBERT (Lan et al.), Romanian BERT (Dumitrescu et al.), and IndoNLU (Wilie et al.).


A Cross-Modal Paradigm

The intuition behind progressive sequence training extends far beyond natural language processing. It operates as a universal training heuristic across diverse modalities:

  • Genomics: Modeling DNA requires processing millions of base pairs at single-nucleotide resolution. In HyenaDNA, Nguyen et al. implemented a curriculum that repeatedly doubled the sequence window size throughout training. This strategy stabilized gradient dynamics and dramatically reduced total training time while scaling to massive genomic contexts.
  • Audio Intelligence: Understanding long-form audio requires moving from immediate acoustic features to global narrative structure. In Audio Flamingo 2, Ghosh et al. utilized a multi-stage curriculum that scaled capabilities from short clips to long-form audio reasoning, allowing the model to smoothly adapt to extended inputs.
  • Computer Vision and Video: When generating or forecasting video dynamics, predicting long temporal horizons directly is highly unstable. As shown by Yang et al., gradually increasing the number of predicted future steps helps networks master immediate motion before tackling long-term trajectory forecasting.
  • Recurrent Neural Networks: This principle even mirrors biological and cognitive constraints. Cristina Savin demonstrated that training recurrent networks on tasks with short temporal delays before gradually increasing the duration of the delay period allows unstructured circuits to successfully organize themselves and bridge long-range temporal dependencies.

Pushing the Boundaries of Context Extension

Beyond initial pretraining, progressive sequence scaling is a critical tool for context window extension—taking an existing model and expanding its capacity to handle millions of tokens.

Rather than jumping directly from a short base context to an extreme target length, state-of-the-art methods employ intermediate, progressive expansion steps. Gao et al. demonstrate that training a model on incrementally longer sequences (e.g., scaling from 64K to 512K tokens) allows the architecture to adapt to increasing positional complexity while maintaining performance across all lengths.

[Base Model: 4K] ──> [Intermediate Stage: 64K] ──> [Target Stage: 512K+]

This progressive stepping strategy ensures that positional embeddings adjust smoothly. For example, Ding et al. leveraged this exact progressive extension strategy in LongRoPE to push large language models to an unprecedented 2 million tokens. By fine-tuning on intermediate lengths first, the model learns to exploit non-uniformities in positional embeddings without suffering from the representational collapse that often occurs during abrupt context expansion.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_curriculum_learning_0d8d515d,
  title = {temporal curriculum learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-curriculum-learning/rdr_0d8d515d408441a991fa4f55897d4d6f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_temporal_curriculum_learning_0d8d515d_v1,
  title = {temporal curriculum learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-curriculum-learning/rdr_0d8d515d408441a991fa4f55897d4d6f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake