Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful paradigm for scaling the reasoning capabilities of Large Language Models (LLMs). By training models on tasks with objective, rule-based feedback—such as mathematical proofs or code execution—algorithms like Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) allow models to discover novel problem-solving paths through trial and error.
However, practitioners frequently encounter a frustrating bottleneck: after an initial surge in performance, the model's learning stalls, and its ability to solve complex problems plateaus or even degrades.
This stagnation is driven by a phenomenon known as policy entropy collapse. During training, the model's output distribution narrows too quickly, causing it to become prematurely overconfident in a small set of familiar trajectories. Understanding why this collapse happens—and how it fundamentally breaks the mechanics of reinforcement learning—is key to unlocking stable, long-term post-training scaling.
The Mechanics of the Collapse
To understand why entropy collapses, we must look at how policy gradient updates interact with the autoregressive nature of language models.
When an LLM explores a reasoning path that successfully triggers a positive reward, the policy gradient objective attempts to maximize the likelihood of that path. It does this by increasing the logits of the tokens generated along that successful trajectory.
Because language models generate text token-by-token, a small, uniform increase in the logits of a few preferred tokens has an exponential effect when passed through the softmax function. As documented by Choshen et al., this creates a "Peakiness Effect," where the probability mass of the entire distribution rapidly concentrates onto the top-1 candidate. RL updates often act less as a tool for discovering new behaviors and more as an aggressive confidence-boosting mechanism for existing biases.
Logit Space: [ token_A: 2.1, token_B: 2.0, token_C: 0.5 ] -> Highly Stochastic
RL Update -> [ token_A: 4.5, token_B: 2.0, token_C: 0.5 ]
Softmax Space: [ token_A: 0.92, token_B: 0.07, token_C: 0.01 ] -> Entropy Collapsed
Once the token-level entropy drops to near-zero levels, the mathematical engine of reinforcement learning breaks down. In a theoretical analysis of LLM-RL algorithms, Han Shen demonstrated that a collapse in policy entropy directly causes the policy gradient to vanish.
Policy gradient methods rely on the variance of the action distribution to determine which direction moves the policy toward higher rewards. When the policy becomes deterministic, this variance drops to zero. The gradient vanishes, the model stops exploring alternative paths, and the entire training process stagnates in a suboptimal local minimum.
Why Entropy Loss Destroys Reasoning
In traditional reinforcement learning environments (like games with fixed action spaces), a deterministic policy is often the ultimate goal. In the context of language and multi-step reasoning, however, a collapsed distribution is catastrophic.
1. Extinguishing "Reasoning Sparks"
Complex reasoning is rarely linear; it requires the model to pivot, re-evaluate, and correct its course mid-generation. Huang et al. identified that standard RL aggressively prunes low-probability transitional tokens—words like "wait", "however", or "alternatively". These tokens serve as cognitive triggers that allow the model to branch into alternative logical paths. When entropy collapses, these "reasoning sparks" are eliminated, forcing the model into rigid, brittle generation patterns. Similarly, Dong et al. observed that aggressive optimization constraints, such as harsh length penalties, accelerate this collapse by suppressing the transitional tokens necessary for complex cognitive steps.
2. Losing the Long Tail of Trajectories
Standard reasoning tasks often feature multiple valid approaches, some of which are rare but essential for handling difficult edge cases. As Bu et al. show, entropy collapse concentrates probability mass exclusively on high-frequency, simple reasoning paths that the model mastered early in training. Consequently, the model loses the behavioral diversity required to navigate the rare, intricate trajectories needed to solve complex problems.
3. The Pass@K Divergence
A clear symptom of entropy collapse is a widening gap between a model's pass@1 accuracy (the probability of getting the right answer on the first try) and its pass@K performance (the probability of finding a correct answer across $K$ independent samples). Ren et al. demonstrate that standard RL with verifiable rewards induces a severe probability concentration effect. While the model might maintain its pass@1 performance by exploiting a single safe path, its pass@K diversity plummets because it can no longer generate alternative valid solutions when its primary strategy fails.
Countermeasures: Preserving the Exploration Space
To prevent training stagnation, recent research focuses on mechanisms that actively defend or dynamically manage the model's exploration space.
Explicit Token-Level Regularization
Instead of relying solely on global KL-divergence penalties against a static reference model, newer approaches directly intervene at the token level.
- Low-probability Regularization (Lp-Reg): Huang et al. introduce a regularization term that explicitly shields low-probability tokens from being pruned, preserving the model's ability to trigger alternative reasoning paths.
- Asymmetric Token Smoothing: Ren et al. propose SimKO, which applies asymmetric token-level regularization. It smooths the probability distribution for correct answers while heavily penalizing the top-1 candidate when the model generates an incorrect answer, forcing the policy to redistribute its mass and explore alternative logical branches.
Rethinking the Optimization Objective
Other methods modify the reward structure or the objective function to avoid rewarding "safe" behaviors too aggressively.
- Risk-Based Optimization: Ren et al. introduce RiskPO, which shifts the focus from optimizing the mean reward to optimizing risk measures on the tail of the reward distribution. This prevents the model from becoming overconfident in paths it has already mastered and keeps the gradient active on challenging edge cases.
- Decoupled Constraints: Zeng et al. utilize decoupled adaptive entropy constraints to dynamically adjust exploration incentives based on task difficulty, ensuring that the model retains high entropy when facing complex, multi-step tool orchestration problems.
Phased Training and Curriculums
Rather than exposing a raw model directly to online RL, researchers have found success by managing the transition of the policy's entropy state over time.
- Behavior Priming: Jin et al. demonstrate that performing Supervised Fine-Tuning (SFT) on core search and verification behaviors before applying RL provides the model with a robust, high-entropy foundation of strategies, preventing it from settling into narrow, suboptimal behaviors.
- Dynamic Curriculum Orchestration: Pang et al. propose orchestrating data delivery based on inference entropy. By continuously prioritizing training samples where the model's current uncertainty is highest, they prevent the uniform confidence sharpening that triggers collapse.