Train Once, Answer All: Many Pretraining Experiments for the Cost of One

Sebastian Bordt, Martin Pawelczyk

Introduction

The gold standard for understanding large language model (LLM) behavior is the controlled pretraining experiment: researchers isolate a single variable—such as data composition or model architecture—train a model from scratch, and observe the results. However, training a competitive LLM incurs massive computational and financial costs.

Because of this "compute tax," most research groups are bottlenecked, forced to test only one hypothesis at a time or rely on small-scale "toy" models. To overcome this limitation, researchers have proposed a new paradigm: "Train Once, Answer All." This approach demonstrates that dozens of independent pretraining experiments can be conducted simultaneously within a single training run without significant interference, effectively slashing the compute cost of LLM research by an order of magnitude.

The "Train Once, Answer All" Paradigm

The core premise of this framework is to leverage the unsupervised, multitask nature of LLMs. Because these models naturally learn disparate concepts simultaneously, researchers can introduce multiple independent interventions into the training data at once.

Comparison of the standard single-experiment paradigm versus the proposed multiple-experiment approach.

To execute this, the authors modified small fractions of the training data (totaling approximately 1.8% of the overall tokens) to serve different experimental goals. These modifications included adding poisoned samples, synthetic math problems, and "canary" strings for privacy testing.

To ensure scientific rigor and prove that one experiment does not confound the results of another, the authors developed Continual Pretraining Dependence Testing (CPDT). CPDT involves taking an intermediate model checkpoint and training it briefly on a high-intensity version of one experiment's data. Researchers then measure if this burst affects the metrics of a different experiment. If the cross-experiment impact is near zero, the interventions are deemed independent.

Additionally, the study introduced a dynamic control algorithm for knowledge acquisition. Rather than exposing a model to a fact a static number of times, the algorithm probes the model's knowledge every 1,000 steps and dynamically adjusts the frequency of that fact in upcoming training batches to guarantee acquisition.

Experimental Setup and Datasets

The framework was validated by performing ten simultaneous experiments during a single 210-billion-token training run, scaling up to models with 2.7 billion parameters. The base pretraining data was OLMo-mix-1124, supplemented with specialized datasets for the embedded experiments:

  • Fictional Knowledge: Realistic but fake entities to track how models acquire new facts.
  • iGSM: Synthetic grade-school math to test reasoning and length generalization.
  • SODA: Dialogue data used to insert "canaries" for privacy and memorization audits.
  • MUSE-News: Utilized for machine unlearning and forgetting experiments.
  • Benchmark Contamination: Ground-truth answers from sets like ARC, BoolQ, and MMLU injected to study test-set overfitting.

Key Findings

1. Successful Replication at a Fraction of the Cost In a single training run, the authors successfully replicated the core findings of five major prior papers covering data contamination, poisoning, memorization, and forgetting.

2. Minimal Interference on Training Dynamics Modifying nearly 2% of the training tokens had almost zero impact on the model’s overall loss, gradient norms, or performance on unrelated benchmarks. The model remained a high-quality generalist despite acting as a testbed for ten different studies simultaneously.

3. Novel Insights into LLM Behavior The simultaneous experiments yielded several new insights:

  • Recency Bias: Using Gaussian watermarking, the authors found that a model's final state is disproportionately influenced by data seen late in the training process.
  • Knowledge Steering: The dynamic control algorithm successfully steered the model to acquire specific knowledge, achieving 25% zero-shot accuracy on targeted facts by the end of training.
  • Length Generalization: Models trained on simple synthetic math problems successfully solved problems requiring more reasoning steps than any seen during training.
  • Scaling Effects: Experimental interventions, such as the impact of data contamination, generally became more pronounced and detectable as model size increased from 179M to 2.7B parameters.

Limitations and Future Directions

While highly effective, the "Train Once, Answer All" approach has boundaries. There is likely a "crowding limit" where too many simultaneous experiments degrade the model's general capabilities. Furthermore, experiments targeting overlapping domains (e.g., two different methods for teaching math) are likely to interfere with one another, complicating the interpretation of results.

Methodologically, the CPDT technique struggled to test for dependencies in verbatim memorization, as that specific experiment relies on the absence of data, which is difficult to simulate in a short continual pretraining burst.

Despite these limitations, this paradigm offers a highly efficient path forward for LLM research. By releasing their Python package and model checkpoints, the authors have provided the community with the tools to conduct rigorous, large-scale scientific experimentation on a realistic compute budget.

Summary updated

Abstract

Recent work has demonstrated that controlled pretraining experiments are a powerful tool for studying the relationship between training data and large language model (LLM) behavior. However, the computational cost of pretraining presents a significant constraint. To overcome this constraint, we propose a new approach where multiple experiments are conducted simultaneously during a *single* training run. We validate our approach by performing ten experiments while training on 210B tokens, with models of up to 2.7B parameters. Although models are trained only once, we can replicate the results of multiple previous works on data contamination, poisoning, and memorization. We also conduct novel investigations into knowledge acquisition, mathematical reasoning, and watermarking. For example, we dynamically update the training data until a model acquires a particular piece of knowledge. Remarkably, the influence of the experiments on the model's training dynamics and overall performance is minimal. However, interactions between experiments may act as a confounder in our approach. We propose continual pretraining dependence testing (CPDT), a novel technique to test for interactions with continual pretraining experiments, finding them to be negligible in our setup. Overall, our results suggest that performing multiple pretraining experiments within a single training run can enable rigorous scientific experimentation with large models on a compute budget.
BibTeX
@inproceedings{bordt2026train,
  author = {Sebastian Bordt and Martin Pawelczyk},
  title = {{Train Once, Answer All: Many Pretraining Experiments for the Cost of One}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=EoBmdFujak},
}
Figure