When training large language or multimodal models to perform complex, multi-step reasoning—such as solving math Olympiad problems, navigating software interfaces, or generating CAD code—standard training recipes often hit a wall. If we rely solely on Supervised Fine-Tuning (SFT), the model becomes a rigid mimic, struggling to generalize to unseen scenarios or recover from its own mistakes. Conversely, if we attempt to train the model via Reinforcement Learning (RL) from scratch, the search space is too vast; the model frequently fails to output valid syntax, resulting in completely sparse rewards.
To resolve this, a consensus has emerged around a two-stage post-training paradigm: first applying SFT to establish structural and syntactic baselines, followed by RL to optimize strategic decision-making, efficiency, and environmental exploration.
Stage 1: Establishing the Structural Scaffold
The primary objective of the first stage is not to maximize the model's ultimate strategic capability, but to build a reliable linguistic and structural scaffold. During this phase, the model undergoes imitation learning on high-quality, structured datasets. As shown by Jiacheng Liu et al., this initial phase teaches the model the basic structure of reasoning before it is optimized for specific task performance.
Specifically, SFT serves as a "cold start" mechanism. For instance, Haiquan Wen et al. demonstrate that initializing with structured reasoning traces prevents model collapse during subsequent policy optimization. This stage forces the model to internalize specific output syntax and formatting constraints—such as reasoning tags or tool-calling protocols—before it is incentivized to explore the environment (Minheng Ni et al.; Zayne Rea Sprague et al.). Without this structural grounding, the infinite prompt or action space makes exploration mathematically intractable (Qiaozhu Mei et al.).
Stage 2: Optimizing Policy and Generalization
Once the model understands how to format its thoughts and execute basic actions, the second stage introduces RL to transition the model from passive token-level mimicry to active policy optimization. In this phase, algorithms like Group Relative Policy Optimization (GRPO) or multi-turn RL are deployed to refine precision, efficiency, and error recovery.
The authors of A Practical Two-Stage Recipe for Mathematical LLMs observe a distinct division of labor: while SFT establishes the model's maximum accuracy potential, RL optimizes the efficiency and reliability of the reasoning process, drastically reducing the computational cost of generating solutions.
Furthermore, RL allows the model to develop exploration capabilities that are absent in static SFT datasets. For example, Haolong Yan et al. demonstrate that moving from SFT to multi-turn RL enables screen-navigation agents to successfully recover from errors and generalize to entirely unseen user interfaces.
Why the Sequence is Non-Negotiable
Why not train both dimensions simultaneously, or reverse the order? The intuition lies in how models handle sparse rewards and statistical calibration.
In tool-augmented or agentic tasks, a model must execute a precise sequence of actions to succeed. If an agent does not know the basic mechanics of a tool, it will receive zero reward during RL exploration. Xuchen Li et al. and Gongrui Zhang et al. show that an initial SFT phase establishes baseline tool-calling syntax, transforming a completely blind RL search into a targeted optimization problem.
Additionally, the two stages target different cognitive dimensions. In the context of medical visual question answering, the authors of CARE note that SFT acts as the primary driver for open-ended generative depth, whereas RL is uniquely suited for refining statistical calibration and eliminating overconfident wrong answers. By separating the acquisition of structural reasoning from the optimization of outcome-based performance, the two-stage recipe ensures that the model remains both highly capable and robustly aligned (Zeming Wei et al.).