RL Excursions during Pre-training: How early is too early for on-policy learning?

RLxF @ ICML 2026 · OpenReview

TL;DR

The standard LLM training recipe treats Reinforcement Learning (RL) as a final "polishing" step, but this paper shows RL is effective much earlier—even before pre-training is finished. By applying RL to intermediate checkpoints, the authors found it can match the performance of the full SFT→RL pipeline, expand (rather than just sharpen) the model's reasoning paths, and avoid the "general knowledge rot" typically caused by supervised fine-tuning.

What Problem Does This Paper Address?

The current "standard recipe" for Large Language Models (LLMs) is a linear pipeline:

  1. Pre-training: Next-token prediction on massive datasets.
  2. Supervised Fine-Tuning (SFT): Learning from high-quality human/AI demonstrations.
  3. Reinforcement Learning (RL): Optimizing for specific rewards (like correct math answers).

The authors question this status quo. Is RL only effective after a model is "fully baked"? Does RL actually teach new reasoning, or does it just "sharpen" the distribution by forcing the model to pick the best of what it already knows? Furthermore, they address the common issue where post-training (specifically SFT) often causes a "regression" in the model’s general, non-specialized capabilities.

Methods

The researchers trained 1B and 4B parameter models from scratch and saved checkpoints at various stages of pre-training. They then branched off from these checkpoints using several strategies:

  • Direct RL: Applying RL (specifically GRPO—Group Relative Policy Optimization) directly to the base model.
  • SFT-Single/Multi: Training on one or many ground-truth solutions.
  • Standard Pipeline: SFT followed by RL.
  • Parallel Averaging (Novel): A new approach that interleaves SFT and RL gradients in a single training step, maintaining independent optimizer states for each to prevent interference.

For the RL reward, they used Verifiable Rewards (RLVR), where the model is rewarded simply for reaching the correct final answer in a math problem, rather than following a specific human-written "trace."

Datasets

  • Pre-training: DOLMino, a 50B-token high-quality subset of the OLMo2 mix (includes Wikipedia, web data, and a significant 20% math component).
  • Post-training: OpenMathInstruct, which provides math problems with multiple reasoning paths.
  • Evaluation:
    • Reasoning: GSM8K (grade-school math) and MATH (competition-level).
  • General Capabilities: A suite of six benchmarks including HellaSwag, ARC, PIQA, and LAMBADA.

Key Insights and Ideas

  • RL is an "Expander," SFT is a "Sharpener": Recent debate suggested RL just narrows a model's focus. This paper finds that when RL is applied to a base model, it actually expands the distribution—the model discovers new reasoning paths it didn't see during pre-training. The "sharpening" effect only happens when RL follows SFT, because SFT has already constrained the model's "imagination."
  • The "SFT Tax": Supervised fine-tuning consistently degrades general model capabilities (a 4–8 percentage point drop). RL, however, leaves these general capabilities almost entirely intact.
  • Data Trumps Scale: For difficult tasks (like the MATH benchmark), simply making the model bigger (4B vs 1B) didn't make RL more effective. However, adding more math-heavy data to the pre-training mix significantly boosted RL's success.
  • Predictive Diagnostics: The authors found that a base model's performance on an 8-shot prompt is a reliable "lightweight diagnostic" for whether RL training will be successful on that checkpoint.

Key Results

  • Early Effectiveness: RL was effective as early as 4B pre-training tokens (well before the model reached "Chinchilla optimality"). On GSM8K, RL boosted pass@1 accuracy from ~2% to ~18% at this early stage.
  • Matching the Pipeline: By 10B tokens, Direct RL matched the performance of the standard SFT→RL pipeline on GSM8K.
  • Parallel Averaging Wins: The proposed Parallel Averaging method outperformed all other recipes that use a single demonstration per problem, achieving a 78.17% pass@32 on GSM8K while preserving the base model's general knowledge.
  • Sparse Data Resilience: RL is significantly more effective than SFT when ground-truth demonstrations are scarce.

Limitations and Future Work

  • Math-Heavy Pre-training: The models were pre-trained on a mix containing 20% math data. It remains to be seen if RL is as effective early on if the pre-training data is a more "standard" web-heavy mix with less reasoning content.
  • Model Scale: While 1B and 4B models are useful for research, the dynamics might shift at the "frontier" scale (70B+ parameters).
  • Algorithm Specificity: The study focused on GRPO. Other RL variants (like those targeting entropy preservation) might interact differently with early pre-training checkpoints.
  • Future Direction: The results suggest we should rethink the entire LLM lifecycle—perhaps RL shouldn't be a "post-training" phase at all, but a core part of the pre-training process itself.

Summary updated

Abstract

The standard LLM training pipeline applies reinforcement learning (RL) only after pre-training and supervised fine-tuning (SFT). We question this status quo by training a LLM from scratch and applying RL, SFT, and SFT followed by RL directly to intermediate pre-training checkpoints. We find that RL is effective very early, and often matches the full SFT$\rightarrow$RL pipeline early as well. Through experiments on harder problems, we find that targeted pre-training data composition is a strong lever for RL effectiveness, even more so than model scale. Beyond reasoning accuracy, applying RL directly to base checkpoints expands the model's distribution; the sharpening effect reported in recent work arises only when RL follows SFT. The general capabilities of the model remain essentially unchanged by RL, while they degrade following SFT. Finally, we merge RL and SFT objectives by \textit{parallel averaging}, which outperforms across all other training methods discussed, across metrics, while preserving general capabilities. Together, these results suggest that LLM training might benefit from an expanded use of RL.
BibTeX
@inproceedings{2026rl,
  title = {{RL Excursions during Pre-training: How early is too early for on-policy learning?}},
  booktitle = {RLxF @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=h7N9pvnwzz},
}
Figure