TL;DR
Controlled pretraining experiments are essential for understanding how training data shapes large language model (LLM) behavior, but their massive computational cost usually limits researchers to one experiment per training run. This paper introduces a "train once, answer all" paradigm that runs multiple independent data interventions simultaneously within a single pretraining run. By modifying less than 2% of the training tokens, the authors successfully conduct ten distinct experiments at once—saving massive amounts of compute with virtually zero impact on overall model performance.
What Problem Does This Paper Address?
To understand the inner workings of LLMs, researchers use controlled pretraining experiments: they systematically alter a small sliver of the training data (e.g., adding corrupted text, specific facts, or math problems) to isolate how those changes impact the final model. Unfortunately, training an LLM from scratch is incredibly expensive. Because of this compute bottleneck, most research projects can only afford to test a single hypothesis per training run, severely restricting the pace of scientific discovery in machine learning.
Methods
The authors propose running multiple independent pretraining experiments simultaneously during a single training run.
To make this setup reliable, they introduce two key methodological innovations:
- Continual Pretraining Dependence Testing (CPDT): A major risk of running multiple experiments at once is cross-contamination (i.e., one experiment interfering with the outcome of another). CPDT acts as a smoke test before the full training run. Researchers take an intermediate model checkpoint, train it briefly on a high-intensity version (~1% of data) of a single experiment's data, and measure the impact on all other experiment metrics to ensure independence.
- Dynamic Control Loops for Knowledge Acquisition: Instead of injecting factual data at a fixed rate, they implement a control algorithm that checks the model's familiarity with a fact every 1,000 steps and dynamically adjusts its frequency in the upcoming data stream to hit a precise target by the end of training.
Datasets
- Base Pretraining Data: The models were trained on 210 billion tokens of
OLMo-mix-1124(which is primarily composed of theDCLM-Baselinedataset). - Experimental Interventions: The authors mixed in small subsets of data from several open datasets, including a fictional knowledge dataset (for factual tracking), the
iGSMsynthetic math dataset (for reasoning), theSODAdialogue dataset (for privacy/canary tracking), andMUSE-News(for machine unlearning evaluation). - Evaluation Suite: Models were evaluated using the
OLMESstandard, which includes benchmarks like ARC-Easy, ARC-Challenge, BoolQ, CSQA, HellaSwag, and MMLU.
Key Insights and Ideas
- Pretraining is an Inherent Multitasker: Because LLMs naturally learn many diverse skills at the same time during pretraining, the model can easily tolerate independent, localized data interventions without hurting its overall development.
- Zero-Interference Between Experiments: Using their CPDT method, the authors discovered that carefully designed controlled experiments show no meaningful cross-dependencies. You truly can get the results of ten independent studies from a single run.
- Benchmarks Interfere, Experiments Don't: Interestingly, CPDT revealed that standard language benchmarks have massive cross-dependencies (e.g., contaminating a model with ARC-Easy data significantly boosts its performance on ARC-Challenge). Controlled experiments, however, remain perfectly isolated.
Key Results
- Five Replications for the Price of One: In a single training run, the authors successfully replicated the conceptual results of five distinct prior papers covering benchmark contamination, canary memorization patterns, prompt-extraction poisoning, verbatim text completion, and batch forgetting curves.
- Negligible Performance Overhead: Modifying 3.7 billion tokens (1.8% of the total data) across 10 experiments had almost no impact on training dynamics. The training loss, validation loss, and weight norms were virtually identical to a completely clean baseline model. Holdout benchmark accuracy only saw a tiny drop from 55.51% to 55.15%.
- Emergent Math Reasoning: In a novel math experiment modifying just 0.09% of the data, the model demonstrated "length-generalization"—solving complex math problems that required more reasoning steps than any problem it saw during training. This capability cleanly emerged once the model scaled to $\ge$ 546 million parameters.
Limitations and Future Work
- The "Absence of Data" Problem: This simultaneous approach struggles with experiments that require completely removing a specific type of data across the entire training run (e.g., testing what happens if a model never sees a specific concept). You cannot easily simulate the total absence of data using a short checkpoint test like CPDT.
- Unknown Upper Bound: While 10 simultaneous experiments worked flawlessly, the authors did not establish the maximum number of interventions a single training run can support before the training dynamics begin to degrade.
- Domain Expansion: It remains to be seen if this multi-experiment framework transfers successfully outside of natural language processing to fields like computer vision or tabular data.
Summary updated