For years, the standard recipe for training capable language models followed a strict hierarchy: first, teach the model to imitate human behavior via Supervised Fine-Tuning (SFT), and second, refine its preferences using Reinforcement Learning (RL). To prevent the model from drifting too far from its human-like starting point, a Kullback-Leibler (KL) divergence penalty was strictly enforced, tethering the active policy to the reference model.
This paradigm is built on two core assumptions: that a model needs human-curated templates to learn how to reason, and that unconstrained exploration will lead to catastrophic policy collapse. However, a growing body of evidence suggests that these constraints actually act as a bottleneck. By applying reinforcement learning directly to raw base models—a paradigm often called "Zero RL"—and completely eliminating the KL divergence penalty, models can discover highly optimized, flexible reasoning paths that human demonstrations actively suppress.
The Bottleneck of Imitation: Why Skip SFT?
When a model undergoes SFT on reasoning chains, it learns to mimic the surface structure of a thought process rather than the underlying logic. Hardy Chen et al. discovered this vulnerability in large vision-language models, demonstrating that SFT can actively undermine subsequent RL by inducing "pseudo-reasoning" patterns. The model learns to output rigid formatting tokens and mimicry patterns without actual comprehension. By skipping SFT and training their model directly through RL, the authors allowed authentic self-correction behaviors and reflective cues to emerge naturally through exploration.
This limitation arises because human demonstrations impose a strict performance ceiling. As Weihao Zeng et al. demonstrate with the SimpleRL-Zoo framework, training "raw" base models directly with rule-based rewards preserves their ability to discover novel reasoning strategies that bypass the constraints of human-demonstrated traces. Similarly, in spatial reasoning tasks like jigsaw puzzles, Zifu Wang et al. found that supervised initialization inadvertently constrains the model's policy, whereas pure RL yields far better generalization to unseen configurations.
Beyond restricting exploration, SFT can also degrade the model's core capabilities. Yu Ran et al. point out that the supervised alignment process often causes "knowledge forgetting," where models lose factual recall while being tuned for human preference. Direct base-model policy optimization avoids this penalty, preserving the model's original parametric knowledge. When human training data is noisy or suboptimal, the authors of another study note that direct RL prevents the model from inheriting human imperfections, allowing it to converge on a more optimal policy than SFT could provide.
Cutting the Tether: Why Eliminate the KL Penalty?
In traditional RLHF, the KL divergence penalty acts as a leash, punishing the model if its token probabilities deviate too far from the initial SFT policy. While this maintains conversational fluency, it cripples a model's ability to solve complex problems.
When solving a mathematical or logical puzzle, the optimal token sequence may look radically different from anything in the base model's original distribution. Jingcheng Hu et al. show that eliminating the KL divergence penalty allows the model to explore the solution space aggressively, facilitating the emergence of novel reasoning strategies. Furthermore, removing this constraint eliminates the need to keep a static reference model in memory, significantly reducing computational and memory overhead.
Traditional RL: [Base Model] <--- KL Leash ---> [Policy Model] ---> Restricted Exploration
Unconstrained: [Base Model] [Policy Model] ---> Aggressive Exploration to Reward
Paradoxically, keeping the KL penalty can also degrade the quality of reasoning text. Lianghuan Huang et al. observed that KL regularization forces models to generate redundant, rambling text or filler phrases simply to satisfy proximity constraints to the base model's distribution. Removing the penalty encourages concise reasoning chains and improves overall task accuracy.
This effect extends beyond text to other generative modalities. In vector graphics generation, Juan A. Rodriguez et al. found that removing the divergence penalty allows models to deviate further from their initial state to discover highly efficient code structures. For text-to-image generation using masked models, Yifu Luo et al. demonstrated that while large models might rely on strict regularization for stability, smaller architectures benefit immensely from the freedom to explore creative solutions, leading to better alignment with complex prompts.
The Mechanics of Unconstrained Exploration
If we remove both the SFT starting point and the KL divergence leash, what stops the model from collapsing into gibberish? The answer lies in a fundamental shift from distribution matching to outcome verification.
Instead of forcing the model to look like a human writer, unconstrained RL relies on precise, verifiable reward signals that evaluate the correctness of the final output. For instance, Xueguang Ma et al. utilize Group Relative Policy Optimization (GRPO) alongside a generative model-based verifier to scale reasoning across diverse domains without any initial SFT traces. In coding tasks, Huaye Zeng et al. leverage automated test-case synthesis to provide exact execution-based rewards, allowing a base model to optimize its existing knowledge directly for task requirements.
When the reward signal is objective and verifiable—such as a compiler error, a math answer key, or a visual rendering engine—the environment itself acts as a natural regularizer. As Xiangxiang Chu et al. demonstrate with Group Policy Gradient (GPG), stripping away critics, reference models, and KL constraints entirely allows the model to directly optimize the true RL objective, consistently outperforming more complex, constrained baselines.
By shifting the optimization target from how the model speaks (imitation) to what the model achieves (verification), unconstrained policy optimization allows complex behaviors like multi-turn tool validation, self-correction, and reflective reasoning to emerge organically from the base weights.
Key Foundational Work
A Deep Reinforcement Learning Approach to First-Order Logic Theorem Proving (2021) — contributed 3 concept elements to this direction.
Effective Reinforcement Learning for Reasoning in Language Models (2025) — contributed 2 concept elements to this direction.
Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation (2025) — contributed 2 concept elements to this direction.