TL;DR
This paper argues that the traditional staged approach to training foundation models—pretraining, fine-tuning, and reinforcement learning (RL)—is algorithmically unnecessary. By framing RL as maximum-likelihood fitting to model-generated data, the authors introduce unified training: a single, continuous learning process that improves compute efficiency, task generalization, and the retention of general capabilities.
What Problem Does This Paper Address?
Building a foundation model currently looks like a relay race. First, you pretrain on massive web crawls using Maximum Likelihood Estimation (MLE). Then, you perform supervised fine-tuning (SFT) on curated demonstrations. Finally, you switch gears entirely to RL algorithms like PPO or GRPO to align the model with rewards.
This "staged" approach creates several headaches:
- Infrastructure Complexity: Engineers must maintain separate, specialized training pipelines for each stage.
- Catastrophic Forgetting: Abruptly shifting from one data source to another often causes the model to lose "general knowledge" acquired in earlier stages.
- Scientific Opacity: It is difficult to study how a decision made during pretraining affects RL performance when the underlying algorithms are different.
The authors ask: Is this separation fundamental, or can we train a model from start to finish with just one algorithm?
Methods
The core breakthrough is a mathematical bridge between RL and MLE. The authors build on Maximum Likelihood Reinforcement Learning (MaxRL), showing that policy gradient updates can be rewritten as MLE updates on a "success-conditioned" distribution of model-generated data.
In this unified framework, the training process is simplified into a single loop:
- The Generator: Samples completions from the current model and weights them based on rewards (e.g., in binary rewards, correct answers get a high weight, incorrect ones get zero).
- The Trainer: Samples from a shared buffer containing both "static" data (pretraining/SFT) and "generated" RL data, applying a single weighted cross-entropy loss.
As shown in Figure 2 (below), the only thing that changes between "stages" is the weighting of tokens in the data format, not the algorithm itself.
Datasets
The authors validated their approach using two main setups:
- Controlled Reasoning: They used "Reasoning Gym" style puzzles (similar to Einstein/Zebra logic puzzles) where they could precisely control difficulty and reasoning patterns. They introduced "Base" (8-9 people) and "Extended" (10-11 people) versions to test generalization.
- Real-World Scaling: They trained a 0.76B parameter model from scratch using ClimbMix (pretraining) and OLMo-3 Dolmino (midtraining), evaluating on standard benchmarks like MATH-500, GSM8K, HumanEval+, MBPP+, and MMLU.
Key Insights and Ideas
- RL is just MLE on the "Right" Data: The most significant insight is that RL isn't a different kind of learning; it's just learning from a distribution that evolves as the model improves.
- Smoothing the Transitions: By mixing midtraining data into the end of pretraining and throughout RL, the authors avoided the "performance dips" typically seen when switching stages.
- Better Data Reuse: Unified training allows the model to revisit static data while learning from its own generations. This "replay" mechanism is built-in rather than being an add-on, which significantly improves the retention of general capabilities.
- Generalization through Diversity: Mixing static oracle-generated reasoning traces with the model's own (sometimes messy) successful generations helped the model generalize better to larger, unseen puzzle sizes than staged training did.
Key Results
- Compute Efficiency: Unified training achieved higher accuracy and task coverage than staged training within the same total FLOP budget (see the right panel of Figure 1).
- Retention: In the 0.76B model experiments, unified training significantly reduced the deterioration of held-out pretraining loss. While staged training often saw a spike in loss (forgetting), unified mixtures kept the loss low while still improving on new tasks.
- Continual Learning: Models trained with the unified approach proved to be better "students" for future tasks. When both unified and staged models were subsequently trained on coding tasks, the unified models reached higher final accuracy.
Limitations and Future Work
While promising, the authors note several areas for further exploration:
- Scale: The experiments were conducted on models up to 0.76B parameters. Testing these principles on 7B+ or 70B+ models is a necessary next step.
- Multi-task Balancing: The current study focused on a few RL tasks at a time. In a production environment, balancing dozens of competing objectives (safety, coding, creative writing, etc.) within one mixture will be more complex.
- Hyperparameter Tuning: The paper leaves open the question of how optimization settings (like learning rate schedules) should be co-designed with the evolving data mixture.
Ultimately, this work suggests that the "science of training trajectories"—deciding what to show a model and when—is more important than inventing new learning algorithms for every stage of development.
Summary updated