Imagine trying to teach someone to play chess by only telling them "you won" or "you lost" at the very end of the game, without ever explaining how the pieces move. If they make entirely random moves, the probability of them accidentally playing a legal game—let alone winning—is practically zero. They will wander endlessly in a massive space of illegal actions, never receiving a single positive reward signal to learn from.
This is the exact challenge encountered when applying Reinforcement Learning (RL) or Preference Fine-Tuning (PFT) directly to base language models for complex, long-horizon tasks. In reasoning, tool use, and multi-step planning, the space of possible token combinations is astronomically large.
To solve this, modern post-training pipelines rely on a supervised warm-start phase (commonly Supervised Fine-Tuning, or SFT) before initiating reinforcement learning. This initialization phase does not necessarily make the model smarter; instead, it acts as a structural anchor that makes subsequent reinforcement learning mathematically and practically viable.
The Format Anchor Problem
Algorithms like Direct Preference Optimization (DPO) or Group Relative Policy Optimization (GRPO) function by evaluating and ranking multiple generated candidates. If a model is asked to solve a complex mathematical problem but has never been trained to structure its thoughts, its outputs will be chaotic, inconsistent, or entirely unstructured.
Without an initial distribution that adheres to basic formatting rules, the reward model or verifier faces a flat, uninformative reward landscape. As Wang et al. observe, base models frequently fail to converge during RL training if they lack an initial supervised phase to establish basic formatting and reasoning structures.
Supervised initialization solves this by narrowing the model's initial output distribution to a coherent "candidate pool." In their study of post-training trade-offs, Kang et al. demonstrate that a small initial phase of supervised fine-tuning serves as a critical formatting anchor. This anchor enables the base model to generate responses that are structured enough for preference-based algorithms to effectively rank and optimize.
[Base Model] ──(Direct RL)──> High Entropy / Exploration Failure (No Reward Signal)
│
(Small SFT Phase)
▼
[Format-Anchored Model] ──(RL / GRPO)──> Targeted Strategy Optimization
Seeding the Action Space with Meta-Tokens
When a model engages in Chain-of-Thought (CoT) reasoning, it is not just generating an answer; it is executing a sequence of internal "meta-actions" such as planning, verifying intermediate steps, and self-correcting.
A raw base model does not inherently understand the functional purpose of allocating test-time compute via specialized reasoning structures or tokens (such as <think> tags). If the model does not know that these tokens can be used to defer its final answer while it processes complex logic, an RL algorithm cannot reward it for doing so.
Shen et al. demonstrate this limitation, showing that reinforcement learning alone is unable to teach a model to utilize novel meta-action tokens from scratch because the base model lacks an initial semantic and formatting grasp of those tokens. Supervised pre-training provides this essential baseline. Rather than directly maximizing final accuracy, the initial SFT phase serves as an inductive prior that surfaces latent reasoning actions and seeds linguistic markers for planning, as highlighted by the authors of OpenVLThinker.
Preventing Training Collapse in Interactive Environments
The necessity of a warm start becomes even more pronounced in multimodal and agentic environments, where models must interact with external tools, web browsers, or simulators. In these domains, a single malformed tool call or syntax error can break the environment loop entirely, resulting in immediate failure.
If an agent cannot successfully execute basic actions, it can never explore long-horizon strategies. Qian et al. show that a preliminary phase of supervised fine-tuning on high-quality interaction trajectories is vital to establish basic tool-use capabilities. Without this cold start, the subsequent RL process often plateaus prematurely or collapses entirely.
Furthermore, purely feedback-driven learning without a supervised baseline can actively damage a model's existing capabilities. Chen et al. discover that attempting to apply Reinforcement Learning from AI Feedback (RLAIF) without a prior SFT stage can degrade performance, underscoring the inherent fragility of optimization processes that lack a grounded starting distribution.
Bootstrapping Advanced Cognitive Behaviors
How do we generate this initial, high-quality distribution if the base model cannot produce it natively? The standard approach is to bootstrap the model using synthetic data generation or rejection fine-tuning driven by a more capable teacher model.
Chen et al. demonstrate this by using a high-capacity teacher to generate high-quality reasoning chains for complex puzzles. This rejection fine-tuning phase ensures the student model can produce coherent thought sequences before it is exposed to the intensive process of learning from verifiable rewards.
Once this baseline is established, RL can begin optimizing for higher-order cognitive strategies that are incredibly difficult to teach via pure imitation. For instance, Yang et al. emphasize that while online RL is excellent at refining advanced behaviors like self-correction, verification, and backtracking, models are generally unable to discover these strategies from scratch if they have not first been exposed to those behavioral patterns during a supervised initialization phase.