Train Once, Answer All: Many Pretraining Experiments for the Cost of One

Sebastian Bordt, Martin Pawelczyk

TL;DR

Controlled pretraining experiments are essential for understanding how training data shapes large language model (LLM) behavior, but their massive computational cost usually limits researchers to one experiment per training run. This paper introduces a "train once, answer all" paradigm that runs multiple independent data interventions simultaneously within a single pretraining run. By modifying less than 2% of the training tokens, the authors successfully conduct ten distinct experiments at once—saving massive amounts of compute with virtually zero impact on overall model performance.

What Problem Does This Paper Address?

To understand the inner workings of LLMs, researchers use controlled pretraining experiments: they systematically alter a small sliver of the training data (e.g., adding corrupted text, specific facts, or math problems) to isolate how those changes impact the final model. Unfortunately, training an LLM from scratch is incredibly expensive. Because of this compute bottleneck, most research projects can only afford to test a single hypothesis per training run, severely restricting the pace of scientific discovery in machine learning.

Methods

The authors propose running multiple independent pretraining experiments simultaneously during a single training run.

Comparison between the standard single-experiment paradigm and the proposed multi-experiment approach.

To make this setup reliable, they introduce two key methodological innovations:

  1. Continual Pretraining Dependence Testing (CPDT): A major risk of running multiple experiments at once is cross-contamination (i.e., one experiment interfering with the outcome of another). CPDT acts as a smoke test before the full training run. Researchers take an intermediate model checkpoint, train it briefly on a high-intensity version (~1% of data) of a single experiment's data, and measure the impact on all other experiment metrics to ensure independence.
  2. Dynamic Control Loops for Knowledge Acquisition: Instead of injecting factual data at a fixed rate, they implement a control algorithm that checks the model's familiarity with a fact every 1,000 steps and dynamically adjusts its frequency in the upcoming data stream to hit a precise target by the end of training.

Datasets

  • Base Pretraining Data: The models were trained on 210 billion tokens of OLMo-mix-1124 (which is primarily composed of the DCLM-Baseline dataset).
  • Experimental Interventions: The authors mixed in small subsets of data from several open datasets, including a fictional knowledge dataset (for factual tracking), the iGSM synthetic math dataset (for reasoning), the SODA dialogue dataset (for privacy/canary tracking), and MUSE-News (for machine unlearning evaluation).
  • Evaluation Suite: Models were evaluated using the OLMES standard, which includes benchmarks like ARC-Easy, ARC-Challenge, BoolQ, CSQA, HellaSwag, and MMLU.

Key Insights and Ideas

  • Pretraining is an Inherent Multitasker: Because LLMs naturally learn many diverse skills at the same time during pretraining, the model can easily tolerate independent, localized data interventions without hurting its overall development.
  • Zero-Interference Between Experiments: Using their CPDT method, the authors discovered that carefully designed controlled experiments show no meaningful cross-dependencies. You truly can get the results of ten independent studies from a single run.
  • Benchmarks Interfere, Experiments Don't: Interestingly, CPDT revealed that standard language benchmarks have massive cross-dependencies (e.g., contaminating a model with ARC-Easy data significantly boosts its performance on ARC-Challenge). Controlled experiments, however, remain perfectly isolated.
Dependence matrices showing that standard benchmarks heavily influence each other (left), while the controlled experiments remain independent (right).

Key Results

  • Five Replications for the Price of One: In a single training run, the authors successfully replicated the conceptual results of five distinct prior papers covering benchmark contamination, canary memorization patterns, prompt-extraction poisoning, verbatim text completion, and batch forgetting curves.
  • Negligible Performance Overhead: Modifying 3.7 billion tokens (1.8% of the total data) across 10 experiments had almost no impact on training dynamics. The training loss, validation loss, and weight norms were virtually identical to a completely clean baseline model. Holdout benchmark accuracy only saw a tiny drop from 55.51% to 55.15%.
  • Emergent Math Reasoning: In a novel math experiment modifying just 0.09% of the data, the model demonstrated "length-generalization"—solving complex math problems that required more reasoning steps than any problem it saw during training. This capability cleanly emerged once the model scaled to $\ge$ 546 million parameters.

Limitations and Future Work

  • The "Absence of Data" Problem: This simultaneous approach struggles with experiments that require completely removing a specific type of data across the entire training run (e.g., testing what happens if a model never sees a specific concept). You cannot easily simulate the total absence of data using a short checkpoint test like CPDT.
  • Unknown Upper Bound: While 10 simultaneous experiments worked flawlessly, the authors did not establish the maximum number of interventions a single training run can support before the training dynamics begin to degrade.
  • Domain Expansion: It remains to be seen if this multi-experiment framework transfers successfully outside of natural language processing to fields like computer vision or tabular data.

Summary updated

Abstract

Recent work has demonstrated that controlled pretraining experiments are a powerful tool for studying the relationship between training data and large language model (LLM) behavior. However, the computational cost of pretraining presents a significant constraint. To overcome this constraint, we propose a new approach where multiple experiments are conducted simultaneously during a single training run. We validate our approach by performing ten experiments while training on 210B tokens, with models of up to 2.7B parameters. Although models are trained only once, we can replicate the results of multiple previous works on data contamination, poisoning, and memorization. We also conduct novel investigations into knowledge acquisition, mathematical reasoning, and watermarking. For example, we dynamically update the training data until a model acquires a particular piece of knowledge. Remarkably, the influence of the experiments on the model's training dynamics and overall performance is minimal. However, interactions between experiments may act as a confounder in our approach. We propose continual pretraining dependence testing (CPDT), a novel technique to test for interactions with continual pretraining experiments, finding them to be negligible in our setup. Overall, our results suggest that performing multiple pretraining experiments within a single training run can enable rigorous scientific experimentation with large models on a compute budget.
BibTeX
@inproceedings{bordt2026train,
  author = {Sebastian Bordt and Martin Pawelczyk},
  title = {{Train Once, Answer All: Many Pretraining Experiments for the Cost of One}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=EoBmdFujak},
}
Figure