TL;DR
The objective clipping mechanism commonly used in PPO and GRPO inherently biases policy entropy, completely independent of the reward signal. Lower clipping (clip-low) pushes entropy up, whereas upper clipping (clip-high) pulls entropy down and typically dominates under standard symmetric settings. By treating clipping bounds as an explicit dial—specifically making clip-low stronger or turning off clip-high—practitioners can prevent "entropy collapse" in RL with verifiable rewards (RLVR) and preserve exploration without hurting reasoning accuracy.
What Problem Does This Paper Address?
Reinforcement learning with verifiable rewards (RLVR) has become the go-to approach for training reasoning models (e.g., DeepSeek-R1). However, RLVR frequently suffers from entropy collapse: policy entropy drops rapidly, forcing the model into near-deterministic responses. This limits exploration and causes reasoning performance to plateau or even degrade on diverse sampling metrics (such as $\text{pass}@k$ for large $k$).
Prior work has attempted to fight entropy collapse using KL-divergence penalties, learning-rate resets, or reward shaping. Yet why entropy decays so aggressively in the first place—and why recent studies observe LLMs improving even with random rewards—remains poorly understood mechanistically.
Methods
The authors investigate the clipping mechanism $[1 - \varepsilon_{\text{low}}, 1 + \varepsilon_{\text{high}}]$ in trajectory-level policy optimization (such as GRPO and PPO):
-
Theoretical Analysis in a Random-Reward Setting:
Assuming zero-mean, symmetric random rewards (which yield zero expected gradient under unclipped REINFORCE), the authors perform a first-order Taylor expansion of policy entropy under both standard Policy Gradient and Natural Policy Gradient (NPG). This analytically isolates the entropy contributions ofclip-low(clipping when advantage $A < 0$) andclip-high(clipping when $A > 0$). -
Empirical Verification & Asymmetric Clipping Control:
Using the open-sourceverlframework, the authors train LLMs under various reward regimes (Bernoulli random rewards, Gaussian random rewards, and verifiable math rewards). They evaluate standard symmetric clipping ($\varepsilon_{\text{low}} = \varepsilon_{\text{high}} = 0.2$) against asymmetric configurations—such as disabling clip-high ($\varepsilon_{\text{high}} = \infty$) while tightening clip-low ($\varepsilon_{\text{low}} \in \{0.1, 0.15, 0.2\}$).
Datasets
The authors use standard, publicly available mathematical reasoning benchmarks (no new datasets were introduced):
- Training: GSM8K and DAPO-Math-17k.
- Evaluation: GSM8K test set, MATH-500, AMC (AI-MO AMC 2023), AIME 2024, and AIME 2025.
Key Insights and Ideas
-
Clipping Inherently Biases Entropy:
The mathematical derivations show thatclip-lowincreases entropy by preventing negative gradient updates from squashing low-probability tokens to zero. In contrast,clip-highdecreases entropy by curtailing positive updates on already likely tokens. -
Standard Symmetric Clipping Defaults to Entropy Decay:
Under standard symmetric thresholds ($\varepsilon_{\text{low}} = \varepsilon_{\text{high}} = 0.2$),clip-highsystematically dominates. As a result, policies experience continuous entropy loss even when rewards are completely random noise. -
De-mystifying "Random Reward" RL:
Recent literature reported curious performance gains when training LLMs with noisy or random rewards. This paper reveals that the clipping objective implicitly runs entropy minimization, explaining why random-reward training sharpens model responses across multiple model families (Qwen, LLaMA, and OLMo) rather than being a model-specific quirk. -
Exploration Can Be Controlled Directly via Clipping:
Instead of adding extra loss terms or heuristic KL warm-starts, adjusting $\varepsilon_{\text{low}}$ and $\varepsilon_{\text{high}}$ directly regulates entropy dynamics.
Key Results
- Validating the Asymmetric Effect: Disabling clip-high ($\varepsilon_{\text{high}} = \infty$, $\varepsilon_{\text{low}} = 0.2$) caused rapid entropy growth in random-reward runs, whereas disabling clip-low ($\varepsilon_{\text{high}} = 0.2$, $\varepsilon_{\text{low}} = 1.0$) caused rapid collapse across Qwen2.5-1.5B, LLaMA-3.2-1B, and OLMo-2-1B.
- Balanced RLVR without Collapse: In true-reward RLVR on GSM8K and DAPO-Math-17k (using Qwen2.5-3B, Qwen2.5-7B, and LLaMA-3-8B), standard clipping caused pass rate metric degradation. In contrast, setting $\varepsilon_{\text{high}} = \infty$ and $\varepsilon_{\text{low}} = 0.15$ successfully stabilized entropy throughout hundreds of training steps.
- Preserved Exploration at Scale: Entropy-controlled models maintained or improved $\text{mean}@8$ / $\text{mean}@32$ accuracy while achieving noticeably higher $\text{pass}@8$ and $\text{pass}@32$ exploration rates on GSM8K, AMC, MATH-500, and AIME (e.g., preventing the steep mid-training drop in $\text{pass}@32$ seen under symmetric clipping).
Limitations and Future Work
- Theoretical Assumptions: The mathematical proofs rely on simplified tabular softmax representations, full-batch gradient descent/NPG, and symmetric random rewards. Real-world RLVR uses deep networks, AdamW, mini-batches, and task-dependent rewards.
- Fine-grained Tuning: While aggressive lower clipping prevents entropy collapse, overly small $\varepsilon_{\text{low}}$ values can trigger entropy explosion.
- Target Entropy vs. Optimal Reasoning: The authors note that while controlled clipping maintains exploration, maximizing ultimate downstream benchmark accuracy may involve more nuanced dynamics than purely maintaining a fixed entropy level.
Summary updated