On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models

Charlie Zhang, Graham Neubig, Xiang Yue

Introduction

The recent success of models like DeepSeek-R1 and OpenAI’s o1 has sparked a fundamental debate in the artificial intelligence community: Does Reinforcement Learning (RL) actually expand a model's reasoning capabilities, or does it simply "sharpen" skills already latent in its pre-training data? A new study from researchers at Carnegie Mellon University provides a definitive look under the hood by using a fully controlled synthetic environment to isolate the causal contributions of every stage in the modern Large Language Model (LLM) training pipeline.

Methodology: The GSM-Infinite Framework

To address the "black box" nature of modern LLM training, the authors developed the GSM-Infinite framework. This system generates reasoning tasks based on Directed Acyclic Graphs (DAGs), allowing for precise control over two critical variables:

  1. Structural Complexity: Varying the number of arithmetic operations to test extrapolative generalization (e.g., solving 15-step problems after training on 10-step problems).
  2. Contextual Flavor: Using different narrative templates (e.g., "zoo animals" vs. "school teachers") to test contextual transfer.

The researchers trained 100M-parameter models through three distinct stages: Pre-training (broad exposure), Mid-training (structured supervised learning), and RL (using Group Relative Policy Optimization, or GRPO). Evaluation was conducted using a process-verified parser that checked every step of the reasoning chain, not just the final answer.

The "1% Seed" and Contextual Generalization

One of the most striking findings of the research is the requirement for a "seed" of exposure during pre-training. For a model to transfer its reasoning logic to a brand-new context via RL, it requires a minimal amount of prior exposure to that context.

Impact of long-tail exposure during pre-training on RL's ability to generalize to new contexts.

As shown in the experimental data, if a context has 0% exposure during pre-training, RL fails to achieve any performance gains regardless of task difficulty. However, even a tiny 1% exposure acts as a catalyst, allowing RL to grow that seed into full mastery. This suggests that while RL is a powerful optimizer, it struggles to bridge completely disjoint contextual gaps without a pre-existing representational anchor.

The "Edge of Competence" and Mid-Training Efficiency

The study identifies a "Goldilocks zone" for RL data: the Edge of Competence. RL is most effective when targeting tasks that are just beyond the model's current reach. If tasks are too easy, RL merely sharpens existing skills; if they are too hard, the model cannot find a successful path, resulting in a zero-reward signal.

Furthermore, the research highlights the critical role of "Mid-training" (continued pre-training on high-quality, structured data). Under a fixed compute budget, allocating resources to a mid-training phase significantly outperforms an RL-only approach.

Comparison of Mid-Training + RL versus Full RL under a fixed training budget.

As illustrated above, the "Mid-Training + RL" strategy yielded an average performance increase of 10.8% over the "Full RL" strategy. This suggests that structured supervised learning acts as a distributional bridge, preparing the model’s internal representations to better utilize the RL objective.

Key Insights and Conclusion

The interplay between training stages reveals three core principles for developing reasoning models:

  • Calibrated Data: RL must target the model's "edge of competence" to drive genuine capability gains.
  • Minimal Contextual Anchors: As little as 1% pre-training exposure is sufficient for RL to enable contextual generalization.
  • Pipeline Synergy: Mid-training is not just an optional step but a compute-efficient necessity that boosts the effectiveness of subsequent RL.

While the study's use of synthetic data and 100M-parameter models provides high internal validity, future work is needed to determine if these scaling laws hold for larger models (7B+) and more diverse reasoning domains like symbolic logic or code generation. Nonetheless, these findings provide a roadmap for more intentional and efficient LLM training recipes.

Summary updated

Abstract

Recent reinforcement learning (RL) techniques have yielded impressive reasoning improvements in language models, yet it remains unclear whether post-training truly extends a model's reasoning ability beyond what it acquires during pre-training. A central challenge is the lack of control in modern training pipelines: large-scale pre-training corpora are opaque, mid-training is often underexamined, and RL objectives interact with unknown prior knowledge in complex ways. To resolve this ambiguity, we develop a fully controlled experimental framework that isolates the causal contributions of pre-training, mid-training, and RL-based post-training. Our approach employs synthetic reasoning tasks with explicit atomic operations, parseable step-by-step reasoning traces, and systematic manipulation of training distributions. We evaluate models along two axes: extrapolative generalization to more complex compositions and contextual generalization across surface contexts. Using this framework, we reconcile competing views on RL's effectiveness. We show that: 1) RL produces true capability gains (pass@128) only when pre-training leaves sufficient headroom and when RL data target the model's edge of competence, tasks at the boundary that are difficult but not yet out of reach. 2) Contextual generalization requires minimal yet sufficient pre-training exposure, after which RL can reliably transfer. 3) Mid-training significantly enhances performance under fixed compute compared with RL only, demonstrating its central but underexplored role in training pipelines. 4) Process-level rewards reduce reward hacking and improve reasoning fidelity. Together, these results clarify the interplay between pre-training, mid-training, and RL, offering a foundation for understanding and improving reasoning LM training strategies.
BibTeX
@inproceedings{zhang2026interplay,
  author = {Charlie Zhang and Graham Neubig and Xiang Yue},
  title = {{On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=TBaUfO9znF},
}
Figure