While Reinforcement Learning (RL) has proven remarkably effective at teaching models to solve complex problems, it faces a fundamental "cold-start" problem: a model cannot be rewarded for a correct reasoning chain if it cannot first generate a coherent one. Recent research across diverse domains—from medical diagnostics to autonomous driving—has converged on a two-stage "R1-style" recipe. First, a Supervised Fine-Tuning (SFT) phase establishes a baseline of syntactic competence and reasoning templates. Second, an RL phase uses those templates to explore, generalize, and self-correct.
The Cold-Start Problem
In pure reinforcement learning, a model explores a state space to find actions that maximize a reward. However, for complex reasoning, the "action space" is the set of all possible strings. If a model has not been trained to follow a specific format (like using <thought> tags), it is unlikely to stumble upon the correct syntax by chance. Without this structure, rule-based reward systems cannot parse the output, leading to a sparse reward signal that prevents learning.
Jinghang Han and colleagues in COPO: Consistency-Aware Policy Optimization argue that SFT is a "mandatory prerequisite" to establish this cold-start capability. Without it, models fail to adhere to the strict formatting requirements necessary for the reward system to function. Similarly, Guangrun Wang found in UML-CoT that robots cannot optimize their underlying logic via RL until they have first learned the formal syntax of the reasoning language through supervised examples.
SFT Sets the Floor, RL Raises the Ceiling
A common intuition in this field is that SFT and RL play complementary roles. As Tianzhe Chu and Saining Xie demonstrate in SFT Memorizes, RL Generalizes, SFT is excellent at helping models memorize specific patterns and formats, while RL enables them to learn generalizable rules.
The SFT phase provides the "scaffolding"—the basic structure of how a thought process should look. Once this scaffolding is in place, RL can then refine the model's behavior. Shiming Yang shows in Demystifying Long Chain-of-Thought Reasoning that initializing with long-form reasoning data creates a significantly higher performance ceiling. Models that start with short, simple reasoning paths often plateau, whereas those with a robust SFT foundation can leverage RL to develop stable, extended reasoning trajectories that actually improve accuracy.
Bootstrapping via Distillation
Where does this high-quality SFT data come from? Often, it is distilled from larger, more capable "teacher" models. Cheng Qian and Heng Ji utilize this in RM-R1, where they initialize reward models with distilled reasoning traces to stabilize the development of deep evaluative capabilities.
In the context of cloud-edge collaboration, Yao Yao and Hai Zhao introduced Guidance-based Knowledge Transfer (GKT), which uses a teacher model to generate the first few tokens of a reasoning path. This direct initialization prevents early-stage hallucinations in smaller models, ensuring they stay on a valid logical path that RL can then optimize.
Preventing Reward Hacking and Hallucination
Beyond establishing a baseline, the supervised "cold-start" phase acts as a guardrail against "reward hacking"—a phenomenon where a model finds a way to get a high reward without actually solving the task. For example, in image editing evaluation, Bowen Qu and Wei Gao found that without a supervised phase, models might learn to generate nonsensical but long justifications just to satisfy length-based rewards (IE-Critic-R1).
In specialized domains like medicine, this grounding is even more critical. Gengliang Li and Bin Li demonstrate in MedFact-R1 that SFT on factual data provides the necessary domain-specific vocabulary. Without this initial knowledge injection, the model lacks the "language" of medicine required to successfully explore and optimize reasoning paths during the RL phase.
Generalizing the Pattern
The effectiveness of this two-stage approach is being validated across increasingly complex modalities:
- Video Reasoning: Ruizhe Chen and colleagues show that SFT using detailed rationales is the foundation that allows RL to succeed in pinpointing events in videos (Datasets and Recipes for Video Temporal Grounding).
- Autonomous Driving: Songyan Zhang and Chen Lv use SFT to ground models in logical patterns before using an LLM-as-critic to refine driving decisions (OpenREAD).
- Tool Use: In DeepEyesV2, researchers found that a supervised cold-start is essential to teach models the basic syntax of tool-calling (like Python or web search) before RL can be used to optimize the strategy of when to use those tools.
Ultimately, the research suggests that while RL is the engine of "thinking" and generalization, SFT is the necessary blueprint. As Chaoxu Pang notes in Reasoning Pattern Matters, the underlying procedural strategy—the pattern—is what allows models to bridge the gap from simple imitation to genuine problem-solving.