Training Foundation Models with a Single Algorithm

Yuda Song, Alvin Zhang, Fahim Tajwar, Daman Arora, Jeff Schneider, Daniel Khashabi, Ruslan Salakhutdinov, J. Andrew Bagnell, Aarti Singh, Andrea ZanettearXiv 2026 · arXiv

TL;DR

This paper argues that the traditional staged approach to training foundation models—pretraining, fine-tuning, and reinforcement learning (RL)—is algorithmically unnecessary. By framing RL as maximum-likelihood fitting to model-generated data, the authors introduce unified training: a single, continuous learning process that improves compute efficiency, task generalization, and the retention of general capabilities.

What Problem Does This Paper Address?

Building a foundation model currently looks like a relay race. First, you pretrain on massive web crawls using Maximum Likelihood Estimation (MLE). Then, you perform supervised fine-tuning (SFT) on curated demonstrations. Finally, you switch gears entirely to RL algorithms like PPO or GRPO to align the model with rewards.

This "staged" approach creates several headaches:

  1. Infrastructure Complexity: Engineers must maintain separate, specialized training pipelines for each stage.
  2. Catastrophic Forgetting: Abruptly shifting from one data source to another often causes the model to lose "general knowledge" acquired in earlier stages.
  3. Scientific Opacity: It is difficult to study how a decision made during pretraining affects RL performance when the underlying algorithms are different.

The authors ask: Is this separation fundamental, or can we train a model from start to finish with just one algorithm?

Methods

The core breakthrough is a mathematical bridge between RL and MLE. The authors build on Maximum Likelihood Reinforcement Learning (MaxRL), showing that policy gradient updates can be rewritten as MLE updates on a "success-conditioned" distribution of model-generated data.

In this unified framework, the training process is simplified into a single loop:

  • The Generator: Samples completions from the current model and weights them based on rewards (e.g., in binary rewards, correct answers get a high weight, incorrect ones get zero).
  • The Trainer: Samples from a shared buffer containing both "static" data (pretraining/SFT) and "generated" RL data, applying a single weighted cross-entropy loss.
Figure 1: The unified training framework (left) and its performance gains over staged training (right).

As shown in Figure 2 (below), the only thing that changes between "stages" is the weighting of tokens in the data format, not the algorithm itself.

Figure 2: Left: the training system. The generator produces RL data from a recent copy of the weights and appends it to the data buffer, which also holds static data; the trainer samples from the buffer and periodically publishes its weights, which the generator loads. Right: the data format. Each cell is a token and the number is its weight in Eq. (2); the stages differ only in the weights.

Datasets

The authors validated their approach using two main setups:

  1. Controlled Reasoning: They used "Reasoning Gym" style puzzles (similar to Einstein/Zebra logic puzzles) where they could precisely control difficulty and reasoning patterns. They introduced "Base" (8-9 people) and "Extended" (10-11 people) versions to test generalization.
  2. Real-World Scaling: They trained a 0.76B parameter model from scratch using ClimbMix (pretraining) and OLMo-3 Dolmino (midtraining), evaluating on standard benchmarks like MATH-500, GSM8K, HumanEval+, MBPP+, and MMLU.

Key Insights and Ideas

  • RL is just MLE on the "Right" Data: The most significant insight is that RL isn't a different kind of learning; it's just learning from a distribution that evolves as the model improves.
  • Smoothing the Transitions: By mixing midtraining data into the end of pretraining and throughout RL, the authors avoided the "performance dips" typically seen when switching stages.
  • Better Data Reuse: Unified training allows the model to revisit static data while learning from its own generations. This "replay" mechanism is built-in rather than being an add-on, which significantly improves the retention of general capabilities.
  • Generalization through Diversity: Mixing static oracle-generated reasoning traces with the model's own (sometimes messy) successful generations helped the model generalize better to larger, unseen puzzle sizes than staged training did.

Key Results

  • Compute Efficiency: Unified training achieved higher accuracy and task coverage than staged training within the same total FLOP budget (see the right panel of Figure 1).
  • Retention: In the 0.76B model experiments, unified training significantly reduced the deterioration of held-out pretraining loss. While staged training often saw a spike in loss (forgetting), unified mixtures kept the loss low while still improving on new tasks.
  • Continual Learning: Models trained with the unified approach proved to be better "students" for future tasks. When both unified and staged models were subsequently trained on coding tasks, the unified models reached higher final accuracy.

Limitations and Future Work

While promising, the authors note several areas for further exploration:

  • Scale: The experiments were conducted on models up to 0.76B parameters. Testing these principles on 7B+ or 70B+ models is a necessary next step.
  • Multi-task Balancing: The current study focused on a few RL tasks at a time. In a production environment, balancing dozens of competing objectives (safety, coding, creative writing, etc.) within one mixture will be more complex.
  • Hyperparameter Tuning: The paper leaves open the question of how optimization settings (like learning rate schedules) should be co-designed with the evolving data mixture.

Ultimately, this work suggests that the "science of training trajectories"—deciding what to show a model and when—is more important than inventing new learning algorithms for every stage of development.

Summary updated

Abstract

Foundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning (SFT), and reinforcement learning (RL), with changes in both data and learning algorithms. We show that this algorithmic separation is not fundamental. By deriving suitable RL objectives as maximum-likelihood fitting to model-generated target distributions, we place all of these stages under a single likelihood-based training algorithm, unified training. This perspective turns the foundation-model training lifecycle into a problem of choosing which data to train on, in what proportions, and when, with conventional staged training as a special case. We instantiate this framework in a unified training system that mixes static data and RL generations within a single trainer. Across controlled and real-world experiments, unified training improves the accuracy-compute tradeoff and task coverage while preserving broader capabilities, and provides a stronger starting point for continual learning. This work lays the groundwork for a unified science of foundation-model training, connecting mathematical principles, training infrastructure, and the development of model capabilities.
BibTeX
@misc{song2026training,
  author = {Yuda Song and Alvin Zhang and Fahim Tajwar and Daman Arora and Jeff Schneider and Daniel Khashabi and Ruslan Salakhutdinov and J. Andrew Bagnell and Aarti Singh and Andrea Zanette},
  title = {{Training Foundation Models with a Single Algorithm}},
  year = {2026},
  eprint = {2610.40564},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.40564},
}
Figure