Abstract
The data used to pretrain large language models has a decisive impact on a model's downstream performance, which has led to a large body of work on data selection methods that aim to automatically determine the most suitable data to use for pretraining. Existing data selection methods suffer from slow and computationally expensive processes, a problem amplified by the increasing size of models and of pretraining datasets. Data mixing, on the other hand, reduces the complexity of data selection by grouping data points together and determining sampling probabilities across entire groups. However, data mixing proportions are typically fixed before training and therefore cannot adapt to changing training dynamics. To address these limitations, we develop an efficient algorithm for Online Data Mixing (ODM) that combines elements from both data selection and data mixing. Based on multi-armed bandit algorithms, our online approach optimizes the data mixing proportions during training. Remarkably, our method trains a model that reaches the final perplexity of the next best method with 19% fewer training iterations, and improves performance on the 5-shot MMLU benchmark by 1.9% relative accuracy, while adding negligible wall-clock time during pretraining.
Introduction
Pre-training a Large Language Model (LLM) requires carefully selecting data from diverse sources such as Wikipedia, GitHub, and academic journals. Traditionally, researchers determine a fixed "recipe" or data mixture before training begins. This static approach has two primary drawbacks. First, a model's learning requirements evolve; a data mixture that is beneficial during the initial stages of training may become redundant as the model matures. Second, current automated optimization methods, such as DoReMi, require training expensive "proxy" models to identify optimal weights, which often fail to transfer across different architectures or tokenizers.
To overcome these challenges, we introduce Online Data Mixing (ODM), a method designed to optimize data proportions dynamically during the primary training run without incurring significant computational overhead.
Methodology: Online Data Mixing via Multi-Armed Bandits
ODM frames the data mixing challenge as a Multi-Armed Bandit (MAB) problem. In this framework, each data domain (e.g., ArXiv, PubMed) represents an "arm" that the algorithm can pull. The core innovation lies in using the training loss as a "free" reward signal. High loss indicates high perplexity, suggesting the model still has significant information to extract from that specific domain.
We adapted the Exp3 (Exponential-weight algorithm for Exploration and Exploitation) algorithm, incorporating a moving average of rewards. Unlike standard bandits that weight all past rewards equally, ODM prioritizes recent rewards to adapt to the model's changing "appetite" as training progresses. Because the loss is already calculated during standard training steps, the computational overhead of ODM is a negligible 0.000007%.
Experimental Setup
We trained our model on The Pile, a diverse 825GB dataset spanning 22 domains. Performance was evaluated using two primary metrics:
- Perplexity: Measured on held-out validation and test sets across all 22 domains.
- Downstream Accuracy: Evaluated via the Massive Multitask Language Understanding (MMLU) benchmark using 5-shot in-context learning across 57 subjects.
Results and Analysis
ODM demonstrates superior efficiency and performance compared to both human-curated weights and state-of-the-art offline methods. As shown in the validation perplexity curves, ODM reaches the final performance level of DoReMi with 19% fewer iterations and outperforms the standard "Pile Weights" baseline 30% faster.
Beyond speed, ODM achieved a final validation perplexity 4.8% lower than the standard Pile-weighted model. On the MMLU benchmark, ODM improved relative accuracy by 3% over the baseline and 1.9% over DoReMi. These results suggest that the optimal data distribution is a moving target; for instance, we observed that the model's need for GitHub data decreased over time while its reliance on academic papers increased. ODM captures these dynamics automatically, whereas static weights calculated by methods like DoReMi were found to be fragile and sensitive to tokenizer changes.
Conclusion
Online Data Mixing provides a highly efficient, adaptive alternative to static data selection. By leveraging training loss as a reward signal, ODM achieves state-of-the-art performance with virtually zero additional cost. While early training stages can exhibit higher perplexity due to batch homogeneity, future work involving a brief "warm-up" period with mixed micro-batches may further stabilize the initial learning phase. Scaling ODM to larger models (70B+) remains a promising avenue for future research.
Summary updated