Efficient Online Data Mixing For Language Model Pre-Training

Alon Albalak, Liangming Pan, Colin Raffel, William Yang Wang

TL;DR

Pre-training large language models typically relies on static, heuristic domain mixtures or computationally heavy auxiliary tuning pipelines like DoReMi. Online Data Mixing (ODM) reformulates domain weighting as a multi-armed bandit problem that adjusts mixture proportions dynamically on the fly using training loss as the reward. With virtually zero extra compute overhead (0.000007%), ODM reaches baseline validation perplexity 19% to 30% faster and achieves higher downstream 5-shot MMLU accuracy.

What Problem Does This Paper Address?

Language model pre-training datasets (such as The Pile) aggregate data from dozens of distinct sources—ranging from academic papers to web scrapes and GitHub code. How much data to sample from each domain is critical to downstream capability.

Historically, practitioners have relied on hand-tuned heuristics or static offline optimization methods like DoReMi. However, existing methods suffer from notable drawbacks:

  1. Static dynamics: Fixed mixtures cannot adapt as the model masters certain domains faster than others during training.
  2. Computational overhead: Approaches like DoReMi require training auxiliary "reference" and "proxy" models to find domain weights.
  3. Poor transferability: Statically discovered weights often fail to transfer across different architectures or tokenizers (e.g., changing from a 256k to a 50k vocabulary).

The authors ask: Can we dynamically optimize domain sampling weights during pre-training without adding significant computational cost or auxiliary training runs?

Methods

ODM casts dynamic domain selection as a Multi-Armed Bandit (MAB) problem:

  • Arms as Domains: Each arm corresponds to one of the $K$ pre-training dataset domains.
  • Exp3 Algorithm with Moving Averages: Instead of the standard Exp3 cumulative reward, ODM tracks an exponential moving average of importance-weighted rewards to prioritize recent performance gains while decaying exploration over time.
  • Reward from Training Loss: Drawing from information theory, perplexity measures model uncertainty and expected information gain from learning the next token. ODM directly uses the forward-pass training loss on a sampled micro-batch as the reward: $$\mathcal{R}_{i,t} = \mathcal{L}(f_\theta, x, y)$$ Because loss is already computed for gradient updates, this reward requires zero extra forward or backward passes.

  • Warmup Schedule: To prevent instability from extreme loss variances in early iterations, sampling weights remain fixed to the initial distribution for the first 1% of training steps.

Datasets

  • Pre-training: The Pile, an 825 GB dataset consisting of 22 distinct domains (e.g., PubMed, ArXiv, Wikipedia, GitHub, StackExchange). Models were pre-trained on 50 billion tokens (1B parameter decoder-only transformer).
  • Evaluation:
  • Domain-specific and average held-out validation and test perplexity across all 22 Pile domains.
  • MMLU (Massive Multitask Language Understanding): Downstream evaluation across 57 multi-choice classification tasks using 5-shot prompts.
  • No new benchmarks were introduced.

Key Insights and Ideas

  • Training Loss as a "Free" Reward: Using batch loss directly aligns optimization with maximal information gain while avoiding auxiliary models or complex scoring functions, adding only 0.000007% wall-clock overhead.
  • Static Weights Do Not Transfer Across Tokenizers: The published DoReMi weights (trained on a 256k vocabulary) performed worse than naive baseline weights when evaluated on a 50k vocabulary tokenizer. Even when recalculated for 50k, DoReMi's minimax objective kept 17 out of 22 domains strictly in the middle of performance rankings.
  • More Uniform is Often Better: ODM naturally shifted away from the skewed human-designed Pile weights toward a more balanced, uniform distribution across domains, while keeping continuous adaptation active through late stages of pre-training.

Key Results

  • Training Efficiency: ODM reached the final validation perplexity of the next-best baseline (DoReMi-50k) with 19% fewer training iterations, and reached the performance of default Pile Weights (TPW) 30% faster.
  • Perplexity Improvements: ODM achieved a final validation perplexity that was:
  • 4.8% lower than The Pile Weights.
  • 2.4% lower than DoReMi-50k.
  • 4.9% lower than DoReMi-256k.
  • ODM achieved the lowest test perplexity on 9 of the 22 domains (compared to 7 for TPW and 3 for DoReMi-50k).
  • Downstream Capabilities (5-shot MMLU):
  • ODM: 28.42%
  • DoReMi-50k: 27.89% (+1.9% relative improvement by ODM)
  • DoReMi-256k: 27.60%
  • The Pile Weights: 27.47% (+3.4% relative improvement by ODM)

Limitations and Future Work

  • Early-stage Micro-batch Homogeneity: Because each gradient accumulation step draws from an individual domain to calculate rewards cleanly, batches within an accumulation window are less diverse than a standard mixed batch. This causes ODM's validation perplexity to start higher than baselines in early training.
  • Within-Batch Domain Mixing: The authors suggest exploring mixed domains within micro-batches during early warmup, switching to pure bandit-selected micro-batches later.
  • Automated Domain Splitting: The study evaluates predefined human domains. Integrating ODM with unsupervised domain clustering algorithms remains an open avenue for fully end-to-end data scheduling.

Summary updated

Abstract

The data used to pretrain large language models has a decisive impact on a model’s downstream performance, which has led to a large body of work on data selection methods that aim to automatically determine the most suitable data to use for pretraining. Existing data selection methods suffer from slow and computationally expensive processes, a problem amplified by the increasing size of models and of pretraining datasets. Data mixing, on the other hand, reduces the complexity of data selection by grouping data points together and determining sampling probabilities across entire groups. However, data mixing proportions are typically fixed before training and therefore cannot adapt to changing training dynamics. To address these limitations, we develop an efficient algorithm for Online Data Mixing (ODM) that combines elements from both data selection and data mixing. Based on multi-armed bandit algorithms, our online approach optimizes the data mixing proportions during training. Remarkably, our method trains a model that reaches the final perplexity of the next best method with 19% fewer training iterations, and improves performance on the 5-shot MMLU benchmark by 1.9% relative accuracy, while adding negligible wall-clock time during pretraining.
BibTeX
@inproceedings{albalak2023efficient,
  author = {Alon Albalak and Liangming Pan and Colin Raffel and William Yang Wang},
  title = {{Efficient Online Data Mixing For Language Model Pre-Training}},
  booktitle = {R0-FoMo @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=9Tze4oy4lw},
}

Loading related content...

Figure