Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models

Jaesung R. Park, Junsu Kim, Gyeongman Kim, Jinyoung Jo, Sean Choi, Jaewoong Cho, Ernest K. RyuSubmission to ICLR 2026 · Withdrawn

TL;DR

The objective clipping mechanism commonly used in PPO and GRPO inherently biases policy entropy, completely independent of the reward signal. Lower clipping (clip-low) pushes entropy up, whereas upper clipping (clip-high) pulls entropy down and typically dominates under standard symmetric settings. By treating clipping bounds as an explicit dial—specifically making clip-low stronger or turning off clip-high—practitioners can prevent "entropy collapse" in RL with verifiable rewards (RLVR) and preserve exploration without hurting reasoning accuracy.

What Problem Does This Paper Address?

Reinforcement learning with verifiable rewards (RLVR) has become the go-to approach for training reasoning models (e.g., DeepSeek-R1). However, RLVR frequently suffers from entropy collapse: policy entropy drops rapidly, forcing the model into near-deterministic responses. This limits exploration and causes reasoning performance to plateau or even degrade on diverse sampling metrics (such as $\text{pass}@k$ for large $k$).

Prior work has attempted to fight entropy collapse using KL-divergence penalties, learning-rate resets, or reward shaping. Yet why entropy decays so aggressively in the first place—and why recent studies observe LLMs improving even with random rewards—remains poorly understood mechanistically.

Methods

The authors investigate the clipping mechanism $[1 - \varepsilon_{\text{low}}, 1 + \varepsilon_{\text{high}}]$ in trajectory-level policy optimization (such as GRPO and PPO):

  1. Theoretical Analysis in a Random-Reward Setting:
    Assuming zero-mean, symmetric random rewards (which yield zero expected gradient under unclipped REINFORCE), the authors perform a first-order Taylor expansion of policy entropy under both standard Policy Gradient and Natural Policy Gradient (NPG). This analytically isolates the entropy contributions of clip-low (clipping when advantage $A < 0$) and clip-high (clipping when $A > 0$).

  2. Empirical Verification & Asymmetric Clipping Control:
    Using the open-source verl framework, the authors train LLMs under various reward regimes (Bernoulli random rewards, Gaussian random rewards, and verifiable math rewards). They evaluate standard symmetric clipping ($\varepsilon_{\text{low}} = \varepsilon_{\text{high}} = 0.2$) against asymmetric configurations—such as disabling clip-high ($\varepsilon_{\text{high}} = \infty$) while tightening clip-low ($\varepsilon_{\text{low}} \in \{0.1, 0.15, 0.2\}$).

Datasets

The authors use standard, publicly available mathematical reasoning benchmarks (no new datasets were introduced):

  • Training: GSM8K and DAPO-Math-17k.
  • Evaluation: GSM8K test set, MATH-500, AMC (AI-MO AMC 2023), AIME 2024, and AIME 2025.

Key Insights and Ideas

  • Clipping Inherently Biases Entropy:
    The mathematical derivations show that clip-low increases entropy by preventing negative gradient updates from squashing low-probability tokens to zero. In contrast, clip-high decreases entropy by curtailing positive updates on already likely tokens.

  • Standard Symmetric Clipping Defaults to Entropy Decay:
    Under standard symmetric thresholds ($\varepsilon_{\text{low}} = \varepsilon_{\text{high}} = 0.2$), clip-high systematically dominates. As a result, policies experience continuous entropy loss even when rewards are completely random noise.

  • De-mystifying "Random Reward" RL:
    Recent literature reported curious performance gains when training LLMs with noisy or random rewards. This paper reveals that the clipping objective implicitly runs entropy minimization, explaining why random-reward training sharpens model responses across multiple model families (Qwen, LLaMA, and OLMo) rather than being a model-specific quirk.

  • Exploration Can Be Controlled Directly via Clipping:
    Instead of adding extra loss terms or heuristic KL warm-starts, adjusting $\varepsilon_{\text{low}}$ and $\varepsilon_{\text{high}}$ directly regulates entropy dynamics.

Key Results

  • Validating the Asymmetric Effect: Disabling clip-high ($\varepsilon_{\text{high}} = \infty$, $\varepsilon_{\text{low}} = 0.2$) caused rapid entropy growth in random-reward runs, whereas disabling clip-low ($\varepsilon_{\text{high}} = 0.2$, $\varepsilon_{\text{low}} = 1.0$) caused rapid collapse across Qwen2.5-1.5B, LLaMA-3.2-1B, and OLMo-2-1B.
  • Balanced RLVR without Collapse: In true-reward RLVR on GSM8K and DAPO-Math-17k (using Qwen2.5-3B, Qwen2.5-7B, and LLaMA-3-8B), standard clipping caused pass rate metric degradation. In contrast, setting $\varepsilon_{\text{high}} = \infty$ and $\varepsilon_{\text{low}} = 0.15$ successfully stabilized entropy throughout hundreds of training steps.
  • Preserved Exploration at Scale: Entropy-controlled models maintained or improved $\text{mean}@8$ / $\text{mean}@32$ accuracy while achieving noticeably higher $\text{pass}@8$ and $\text{pass}@32$ exploration rates on GSM8K, AMC, MATH-500, and AIME (e.g., preventing the steep mid-training drop in $\text{pass}@32$ seen under symmetric clipping).

Limitations and Future Work

  • Theoretical Assumptions: The mathematical proofs rely on simplified tabular softmax representations, full-batch gradient descent/NPG, and symmetric random rewards. Real-world RLVR uses deep networks, AdamW, mini-batches, and task-dependent rewards.
  • Fine-grained Tuning: While aggressive lower clipping prevents entropy collapse, overly small $\varepsilon_{\text{low}}$ values can trigger entropy explosion.
  • Target Entropy vs. Optimal Reasoning: The authors note that while controlled clipping maintains exploration, maximizing ultimate downstream benchmark accuracy may involve more nuanced dynamics than purely maintaining a fixed entropy level.

Summary updated

Abstract

Reinforcement learning with verifiable rewards (RLVR) has recently emerged as the leading approach for enhancing the reasoning capabilities of large language models (LLMs). However, RLVR is prone to entropy collapse, where the LLM quickly converges to a near-deterministic form, hindering exploration and progress during prolonged RL training. In this work, we reveal that the clipping mechanism in PPO and GRPO induces biases on entropy. Through theoretical and empirical analyses, we show that clip-low increases entropy, while clip-high decreases it. Further, under standard clipping parameters, the effect of clip-high dominates, resulting in an overall entropy reduction even when purely random rewards are provided to the RL algorithm. Our findings highlight an overlooked confounding factor in RLVR: independent of the reward signal, the clipping mechanism influences entropy, which in turn affects the reasoning behavior. Furthermore, our analysis demonstrates that clipping can be deliberately used to control entropy. Specifically, with a more aggressive clip-low value, one can increase entropy, promote exploration, and ultimately prevent entropy collapse in RLVR training.
BibTeX
@misc{park2025clip,
  author = {Jaesung R. Park and Junsu Kim and Gyeongman Kim and Jinyoung Jo and Sean Choi and Jaewoong Cho and Ernest K. Ryu},
  title = {{Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models}},
  year = {2025},
  eprint = {2509.26114},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.26114},
}
Figure