Train Once, Answer All: Many Pretraining Experiments for the Cost of One

Sebastian Bordt, Martin Pawelczyk

Abstract

Recent work has demonstrated that controlled pretraining experiments are a powerful tool for understanding learning, reasoning, and memorization in large language models (LLMs). However, the computational cost of pretraining presents a significant constraint. To overcome this constraint, we propose to conduct multiple pretraining experiments simultaneously during a single training run. We demonstrate the feasibility of this approach by conducting ten experiments during the training of a 1.5B parameter model on 210B tokens. Although we only train a single model, we can replicate the results from multiple previous works on data contamination, poisoning, and memorization. We also conduct novel investigations into knowledge acquisition, mathematical reasoning, and watermarking. For example, we dynamically update the training data until the model acquires a particular piece of knowledge. Remarkably, the influence of the ten experiments on the model's training dynamics and overall performance is minimal. However, interactions between different experiments may act as a potential confounder in our approach. We propose to test for interactions with continual pretraining experiments, finding them to be negligible in our setup. Overall, our findings suggest that performing multiple pretraining experiments in a single training run can enable rigorous scientific experimentation with large models on a compute budget.

Introduction

In the current landscape of Large Language Model (LLM) development, controlled pretraining experiments—where specific variables such as data composition or architectural parameters are isolated and tested from scratch—represent the gold standard for scientific inquiry. However, the prohibitive cost of training multi-billion parameter models often restricts this level of rigor to well-funded industrial labs. For most researchers, particularly in academia, the "compute tax" forces a choice between testing a single hypothesis or foregoing pretraining experiments entirely.

This constraint has fostered a "one-shot" research culture, where decisions are based on intuition or fine-tuning results that may not accurately reflect the dynamics of the pretraining phase. To address this bottleneck, we introduce the "Train Once, Answer All" framework. This approach enables the simultaneous execution of dozens of independent pretraining experiments within a single training run, effectively reducing the cost of scientific discovery by an order of magnitude.

The "Train Once, Answer All" Framework

The core of our proposal is a paradigm shift from the standard single-experiment model to a multi-intervention approach. By carefully injecting small amounts of experimental data into a larger pretraining corpus, we can observe the model's reaction to various stimuli without compromising its general capabilities.

Comparison between the standard single-experiment paradigm and the proposed multi-experiment framework.

Our methodology relies on three key technical pillars:

  1. Experimental Injection: We modify a minimal fraction (approximately 1.8%) of the total pretraining tokens. This data includes "canaries" for memorization tests, poisoned samples for safety audits, and specific knowledge probes.
  2. Continual Pretraining Dependence Testing (CPDT): To ensure that multiple experiments do not interfere with one another, we developed CPDT. This "pre-flight check" involves taking an intermediate checkpoint and running high-intensity versions of each experiment. By measuring cross-experiment influence, we can quantify dependencies and ensure experimental independence.
  3. Dynamic Knowledge Control: For experiments involving knowledge acquisition, we employ a control algorithm that monitors the model's "knowledge likelihood" every 1,000 steps. The frequency of relevant training data is then adjusted dynamically to reach specific learning targets.

Experimental Setup and Datasets

We validated our framework by training an OLMo-2 model (ranging from 179M to 2.7B parameters) on 210 billion tokens. The base training data was derived from the OLMo-mix-1124, primarily utilizing the DCLM-Baseline.

The experimental interventions utilized several specialized datasets:

  • Fictional Knowledge: Realistic but fictitious entities used to track the speed and depth of knowledge acquisition.
  • iGSM: A synthetic grade-school math reasoning dataset designed to test logical progression.
  • SODA: A million-scale dialogue dataset used to audit privacy and memorization.
  • MUSE-News: Utilized for experiments regarding unlearning and data provenance.

To ensure the model remained a healthy, general-purpose LLM, we evaluated it against the OLMES standard suite, including benchmarks like ARC, BoolQ, and MMLU.

Key Findings and Results

Our experiments yielded several significant insights into the nature of LLM pretraining:

Robustness of Pretraining: Perhaps the most striking result is that injecting 3.7 billion modified tokens (1.8% of the total) had a negligible impact on the model’s overall training dynamics. The loss curves and gradient norms of the experimental model remained nearly identical to a clean baseline, suggesting that pretraining is a robustly multitask process.

Successful Replication: We successfully replicated the findings of five major previous studies in a single run. These replications covered critical areas such as data contamination, poisoning, and memorization, proving that our framework can replace multiple independent runs.

Emergent Reasoning: By replacing only 0.09% of the training data with synthetic math problems (iGSM), we observed a massive jump in accuracy. Furthermore, the model demonstrated "length generalization," solving problems more complex than those encountered during training. This capability was found to be scale-dependent, only emerging significantly in models with 546M parameters or more.

Privacy and Watermarking: Our results indicate that Gaussian Watermarks are far more effective for auditing data provenance than traditional "rare token" canaries, particularly when data repetition is low. We also identified a "recency bias," where the model's final state is disproportionately influenced by data seen late in the training process.

Conclusion and Future Work

The "Train Once, Answer All" framework demonstrates that rigorous scientific experimentation on large models is possible even on a limited compute budget. By treating pretraining as a high-capacity channel for multiple simultaneous interventions, researchers can gain broader insights without the linear increase in cost.

However, certain limitations remain. This approach is best suited for experiments requiring small data modifications; interventions requiring a majority of the training data would likely cause interference. Future work should explore the upper bounds of how many experiments can be packed into a single run and whether these findings extend to non-textual domains like computer vision.

Summary updated

Abstract

Recent work has demonstrated that controlled pretraining experiments are a powerful tool for understanding learning, reasoning, and memorization in large language models (LLMs). However, the computational cost of pretraining presents a significant constraint. To overcome this constraint, we propose to conduct multiple pretraining experiments simultaneously during a single training run. We demonstrate the feasibility of this approach by conducting ten experiments during the training of a 1.5B parameter model on 210B tokens. Although we only train a single model, we can replicate the results from multiple previous works on data contamination, poisoning, and memorization. We also conduct novel investigations into knowledge acquisition, mathematical reasoning, and watermarking. For example, we dynamically update the training data until the model acquires a particular piece of knowledge. Remarkably, the influence of the ten experiments on the model's training dynamics and overall performance is minimal. However, interactions between different experiments may act as a potential confounder in our approach. We propose to test for interactions with continual pretraining experiments, finding them to be negligible in our setup. Overall, our findings suggest that performing multiple pretraining experiments in a single training run can enable rigorous scientific experimentation with large models on a compute budget.
BibTeX
@inproceedings{bordt2026train,
  author = {Sebastian Bordt and Martin Pawelczyk},
  title = {{Train Once, Answer All: Many Pretraining Experiments for the Cost of One}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=EoBmdFujak},
}
Figure