Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for enhancing Large Language Model (LLM) reasoning capabilities. However, it often suffers from policy entropy collapse, where models prematurely converge on narrow reasoning paths, stifling exploration. This paper introduces Adaptive Entropy Regularization (AER), a framework that dynamically adjusts exploration based on task difficulty and the model's current state. By implementing difficulty-aware allocation, initial-anchored targets, and dynamic global adjustments, AER significantly improves reasoning performance and training stability across rigorous mathematical benchmarks.
Introduction: The Challenge of Policy Entropy Collapse
In the development of reasoning-capable LLMs, such as OpenAI-o1 and DeepSeek-R1, RLVR is a critical tool. Yet, training is notoriously brittle due to policy entropy collapse. When a model becomes overconfident too early, it stops exploring the broader solution space, leading to suboptimal performance. While traditional entropy regularization adds a bonus for uncertainty, finding a fixed coefficient is difficult: too high leads to "entropy explosion" (gibberish output), while too low fails to prevent collapse. The authors argue that exploration must be dynamic to account for varying task difficulties and model capacities.
Adaptive Entropy Regularization (AER)
The authors propose AER to replace static coefficients with a three-part adaptive controller designed to maintain an optimal exploration-exploitation balance.
Core Components
- Difficulty-Aware Coefficient Allocation (C1): AER estimates task difficulty using the model's current group accuracy. Struggling models receive a higher entropy bonus to encourage exploration, while successful models receive less to allow for fine-tuning.
- Initial-Anchored Target Entropy (C2): Instead of arbitrary targets, AER sets a target entropy ($H^*$) as a fraction of the model's initial entropy ($\hat{H}_0$). This ensures a consistent exploration budget regardless of the base model's starting point or sampling temperature.
- Dynamic Global Coefficient Adjustment (C3): A closed-loop feedback system monitors actual policy entropy. If it falls below the target, the global scaling factor ($\alpha$) increases; if it rises too high, $\alpha$ decreases, preventing both collapse and explosion.
Empirical Motivation and Insights
Preliminary analysis reveals that the "sweet spot" for exploration varies significantly by task difficulty and model state.
Key insights from the study include:
- Task-Specific Needs: Increasing entropy bonuses helps on hard datasets (like AIME) but can degrade performance on easy ones (like GSM8K) by introducing unnecessary randomness. This justifies the difficulty-aware component of AER.
- Exploration vs. Verbosity: AER maintains higher diversity without increasing verbosity. In fact, it often results in shorter average response lengths, suggesting the model finds more efficient reasoning paths rather than "rambling" to maximize entropy.
- The Sweet Spot: Effective exploration occurs slightly below the model's initial entropy, where the model remains flexible enough to learn without losing its way.
Key Results
AER was evaluated on rigorous benchmarks including AIME 2024/2025, AMC 2023, and MATH500 using the Group Relative Policy Optimization (GRPO) algorithm.
- Performance Gains: On the Qwen3-4B-Base model, AER achieved a +7.2% average improvement in pass@1 over vanilla GRPO.
- Enhanced Exploration: Pass@k metrics showed significant gains; for AIME2025, AER's pass@256 reached 62.8%, compared to 47.1% for vanilla GRPO.
- Stability and Generalization: AER maintained a steady state throughout training without the crashes common in fixed-coefficient runs and generalized across different architectures (Qwen2.5 vs. Qwen3) without hyperparameter retuning.
Conclusion and Future Work
AER demonstrates that entropy regularization, when made adaptive, is a potent tool for stabilizing LLM reinforcement learning. By anchoring exploration to initial model states and task difficulty, it unlocks significant reasoning potential. Future work includes scaling the framework to 70B+ models and extending difficulty estimation to non-verifiable domains like creative writing or general chat.
Summary updated