Efficient Online Data Mixing For Language Model Pre-Training

Alon Albalak, Liangming Pan, Colin Raffel, William Yang Wang

Abstract

The data used to pretrain large language models has a decisive impact on a model's downstream performance, which has led to a large body of work on data selection methods that aim to automatically determine the most suitable data to use for pretraining. Existing data selection methods suffer from slow and computationally expensive processes, a problem amplified by the increasing size of models and of pretraining datasets. Data mixing, on the other hand, reduces the complexity of data selection by grouping data points together and determining sampling probabilities across entire groups. However, data mixing proportions are typically fixed before training and therefore cannot adapt to changing training dynamics. To address these limitations, we develop an efficient algorithm for Online Data Mixing (ODM) that combines elements from both data selection and data mixing. Based on multi-armed bandit algorithms, our online approach optimizes the data mixing proportions during training. Remarkably, our method trains a model that reaches the final perplexity of the next best method with 19% fewer training iterations, and improves performance on the 5-shot MMLU benchmark by 1.9% relative accuracy, while adding negligible wall-clock time during pretraining.

Introduction

Pre-training a Large Language Model (LLM) requires carefully selecting data from diverse sources such as Wikipedia, GitHub, and academic journals. Traditionally, researchers determine a fixed "recipe" or data mixture before training begins. This static approach has two primary drawbacks. First, a model's learning requirements evolve; a data mixture that is beneficial during the initial stages of training may become redundant as the model matures. Second, current automated optimization methods, such as DoReMi, require training expensive "proxy" models to identify optimal weights, which often fail to transfer across different architectures or tokenizers.

To overcome these challenges, we introduce Online Data Mixing (ODM), a method designed to optimize data proportions dynamically during the primary training run without incurring significant computational overhead.

Methodology: Online Data Mixing via Multi-Armed Bandits

ODM frames the data mixing challenge as a Multi-Armed Bandit (MAB) problem. In this framework, each data domain (e.g., ArXiv, PubMed) represents an "arm" that the algorithm can pull. The core innovation lies in using the training loss as a "free" reward signal. High loss indicates high perplexity, suggesting the model still has significant information to extract from that specific domain.

The Online Data Mixing (ODM) workflow using a Multi-Armed Bandit approach to update data mixing proportions dynamically.

We adapted the Exp3 (Exponential-weight algorithm for Exploration and Exploitation) algorithm, incorporating a moving average of rewards. Unlike standard bandits that weight all past rewards equally, ODM prioritizes recent rewards to adapt to the model's changing "appetite" as training progresses. Because the loss is already calculated during standard training steps, the computational overhead of ODM is a negligible 0.000007%.

Experimental Setup

We trained our model on The Pile, a diverse 825GB dataset spanning 22 domains. Performance was evaluated using two primary metrics:

  1. Perplexity: Measured on held-out validation and test sets across all 22 domains.
  2. Downstream Accuracy: Evaluated via the Massive Multitask Language Understanding (MMLU) benchmark using 5-shot in-context learning across 57 subjects.

Results and Analysis

ODM demonstrates superior efficiency and performance compared to both human-curated weights and state-of-the-art offline methods. As shown in the validation perplexity curves, ODM reaches the final performance level of DoReMi with 19% fewer iterations and outperforms the standard "Pile Weights" baseline 30% faster.

Validation perplexity over training steps, illustrating that ODM reaches target performance significantly faster than baselines.

Beyond speed, ODM achieved a final validation perplexity 4.8% lower than the standard Pile-weighted model. On the MMLU benchmark, ODM improved relative accuracy by 3% over the baseline and 1.9% over DoReMi. These results suggest that the optimal data distribution is a moving target; for instance, we observed that the model's need for GitHub data decreased over time while its reliance on academic papers increased. ODM captures these dynamics automatically, whereas static weights calculated by methods like DoReMi were found to be fragile and sensitive to tokenizer changes.

Conclusion

Online Data Mixing provides a highly efficient, adaptive alternative to static data selection. By leveraging training loss as a reward signal, ODM achieves state-of-the-art performance with virtually zero additional cost. While early training stages can exhibit higher perplexity due to batch homogeneity, future work involving a brief "warm-up" period with mixed micro-batches may further stabilize the initial learning phase. Scaling ODM to larger models (70B+) remains a promising avenue for future research.

Summary updated

Abstract

The data used to pretrain large language models has a decisive impact on a model's downstream performance, which has led to a large body of work on data selection methods that aim to automatically determine the most suitable data to use for pretraining. Existing data selection methods suffer from slow and computationally expensive processes, a problem amplified by the increasing size of models and of pretraining datasets. Data mixing, on the other hand, reduces the complexity of data selection by grouping data points together and determining sampling probabilities across entire groups. However, data mixing proportions are typically fixed before training and therefore cannot adapt to changing training dynamics. To address these limitations, we develop an efficient algorithm for Online Data Mixing (ODM) that combines elements from both data selection and data mixing. Based on multi-armed bandit algorithms, our online approach optimizes the data mixing proportions during training. Remarkably, our method trains a model that reaches the final perplexity of the next best method with 19\% fewer training iterations, and improves performance on the 5-shot MMLU benchmark by 1.9% relative accuracy, while adding negligible wall-clock time during pretraining.
BibTeX
@inproceedings{albalak2023efficient,
  author = {Alon Albalak and Liangming Pan and Colin Raffel and William Yang Wang},
  title = {{Efficient Online Data Mixing For Language Model Pre-Training}},
  booktitle = {R0-FoMo @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=9Tze4oy4lw},
}
Figure