Rediscovering Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning

Xiaoyun Zhang, Xiaojian Yuan, Di Huang, Wang You, Chen Hu, Jingqing Ruan, Kejiang Chen, Xing Hu

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for enhancing Large Language Model (LLM) reasoning capabilities. However, it often suffers from policy entropy collapse, where models prematurely converge on narrow reasoning paths, stifling exploration. This paper introduces Adaptive Entropy Regularization (AER), a framework that dynamically adjusts exploration based on task difficulty and the model's current state. By implementing difficulty-aware allocation, initial-anchored targets, and dynamic global adjustments, AER significantly improves reasoning performance and training stability across rigorous mathematical benchmarks.

Introduction: The Challenge of Policy Entropy Collapse

In the development of reasoning-capable LLMs, such as OpenAI-o1 and DeepSeek-R1, RLVR is a critical tool. Yet, training is notoriously brittle due to policy entropy collapse. When a model becomes overconfident too early, it stops exploring the broader solution space, leading to suboptimal performance. While traditional entropy regularization adds a bonus for uncertainty, finding a fixed coefficient is difficult: too high leads to "entropy explosion" (gibberish output), while too low fails to prevent collapse. The authors argue that exploration must be dynamic to account for varying task difficulties and model capacities.

Adaptive Entropy Regularization (AER)

The authors propose AER to replace static coefficients with a three-part adaptive controller designed to maintain an optimal exploration-exploitation balance.

Overview of the AER framework compared to fixed entropy regularizers, highlighting the three core components: Difficulty-Aware Allocation (C1), Initial-Anchored Target Entropy (C2), and Dynamic Global Adjustment (C3).

Core Components

  1. Difficulty-Aware Coefficient Allocation (C1): AER estimates task difficulty using the model's current group accuracy. Struggling models receive a higher entropy bonus to encourage exploration, while successful models receive less to allow for fine-tuning.
  2. Initial-Anchored Target Entropy (C2): Instead of arbitrary targets, AER sets a target entropy ($H^*$) as a fraction of the model's initial entropy ($\hat{H}_0$). This ensures a consistent exploration budget regardless of the base model's starting point or sampling temperature.
  3. Dynamic Global Coefficient Adjustment (C3): A closed-loop feedback system monitors actual policy entropy. If it falls below the target, the global scaling factor ($\alpha$) increases; if it rises too high, $\alpha$ decreases, preventing both collapse and explosion.

Empirical Motivation and Insights

Preliminary analysis reveals that the "sweet spot" for exploration varies significantly by task difficulty and model state.

Analysis of entropy coefficients across easy and hard datasets, showing the impact on accuracy and token length, alongside initial entropy variations across models.

Key insights from the study include:

  • Task-Specific Needs: Increasing entropy bonuses helps on hard datasets (like AIME) but can degrade performance on easy ones (like GSM8K) by introducing unnecessary randomness. This justifies the difficulty-aware component of AER.
  • Exploration vs. Verbosity: AER maintains higher diversity without increasing verbosity. In fact, it often results in shorter average response lengths, suggesting the model finds more efficient reasoning paths rather than "rambling" to maximize entropy.
  • The Sweet Spot: Effective exploration occurs slightly below the model's initial entropy, where the model remains flexible enough to learn without losing its way.

Key Results

AER was evaluated on rigorous benchmarks including AIME 2024/2025, AMC 2023, and MATH500 using the Group Relative Policy Optimization (GRPO) algorithm.

  • Performance Gains: On the Qwen3-4B-Base model, AER achieved a +7.2% average improvement in pass@1 over vanilla GRPO.
  • Enhanced Exploration: Pass@k metrics showed significant gains; for AIME2025, AER's pass@256 reached 62.8%, compared to 47.1% for vanilla GRPO.
  • Stability and Generalization: AER maintained a steady state throughout training without the crashes common in fixed-coefficient runs and generalized across different architectures (Qwen2.5 vs. Qwen3) without hyperparameter retuning.

Conclusion and Future Work

AER demonstrates that entropy regularization, when made adaptive, is a potent tool for stabilizing LLM reinforcement learning. By anchoring exploration to initial model states and task difficulty, it unlocks significant reasoning potential. Future work includes scaling the framework to 70B+ models and extending difficulty estimation to non-verifiable domains like creative writing or general chat.

Summary updated

Abstract

Reasoning ability has become a defining capability of Large Language Models (LLMs), with Reinforcement Learning with Verifiable Rewards (RLVR) emerging as a key paradigm to enhance it. However, RLVR training often suffers from policy entropy collapse, where the policy becomes overly deterministic, hindering exploration and limiting reasoning performance. While entropy regularization is a common remedy, its effectiveness is highly sensitive to the fixed coefficient, making it unstable across tasks and models. In this work, we revisit entropy regularization in RLVR and argue that its potential has been largely underestimated. Our analysis shows that (i) tasks of varying difficulty demand distinct exploration intensities, and (ii) balanced exploration may require the policy entropy to be maintained within a moderate range below its initial level. Therefore, we propose Adaptive Entropy Regularization (AER)--a framework that dynamically balances exploration and exploitation via three components: difficulty-aware coefficient allocation, initial-anchored target entropy, and dynamic global coefficient adjustment. Experiments on multiple mathematical reasoning benchmarks show that AER consistently outperforms baselines, improving both reasoning accuracy and exploration capability.
BibTeX
@misc{zhang2025rediscovering,
  author = {Xiaoyun Zhang and Xiaojian Yuan and Di Huang and Wang You and Chen Hu and Jingqing Ruan and Kejiang Chen and Xing Hu},
  title = {{Rediscovering Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning}},
  year = {2025},
  eprint = {2510.10959},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.10959},
}
Figure