TL;DR
This paper introduces Dynamic Hybrid Policy Optimization (DHPO), a novel reinforcement learning approach for optimizing large language models in reasoning tasks. DHPO effectively combines token-level and sequence-level learning signals, overcoming the limitations of existing methods that focus on only one granularity, leading to more stable and accurate performance on mathematical reasoning benchmarks.
What Problem Does This Paper Address?
Large language models (LLMs) are increasingly being used for complex reasoning tasks, such as mathematics and programming, where solutions can be automatically verified. Reinforcement Learning with Verifiable Rewards (RLVR) is a promising paradigm for training these models. However, existing RLVR methods face a trade-off:
- Group Relative Policy Optimization (GRPO) uses token-level importance ratios, which allows for fine-grained credit assignment but often leads to high variance and unstable training.
- Group Sequence Policy Optimization (GSPO) uses sequence-level importance ratios, which provides more stable training by aligning with sequence-level rewards but sacrifices the ability to assign credit to individual tokens.
This dichotomy leaves a gap in achieving both robust stability and precise credit assignment, which is crucial for complex reasoning.
Methods
The core innovation is Dynamic Hybrid Policy Optimization (DHPO), which unifies GRPO and GSPO by using a hybrid importance ratio. This ratio is a weighted combination of token-level and sequence-level importance ratios.
Key components of DHPO include:
- Hybrid Importance Ratio: Instead of a single ratio, DHPO uses a weighted sum: $m_{i,t}(\theta) = w_{i,t} r_{i,t}(\theta) + (1 - w_{i,t}) s_{i,t}(\theta)$. This allows for a smooth interpolation between token-wise and sequence-wise updates.
- Two Weighting Strategies:
- Averaged Mixing: A simple, fixed weight ($w_{i,t} = 0.5$) that averages the token and sequence signals.
- Entropy-guided Mixing: A dynamic weight that adjusts based on the policy's local uncertainty (token entropy). Higher uncertainty leads to more weight on token-level ratios (for exploration), while higher confidence shifts weight to sequence-level ratios (for stability).
- Branch-Specific Clipping: To further stabilize training, DHPO applies clipping independently to the token-level and sequence-level ratios before they are mixed. This prevents outliers in one branch from dominating the update and ensures each component stays within its own trust region.
Datasets
The paper evaluates DHPO on a suite of mathematical reasoning benchmarks using the SimpleRL framework. These include:
- AIME 2024/2025
- AMC 2023
- OlympiadBench
- MATH-500
- Minerva Math
- GSM8K
The models used for training and evaluation are from the Qwen3 series (1.7B, 4B, and 30B-A3B-Base).
Key Insights and Ideas
- Bridging the Granularity Gap: The fundamental insight is that neither purely token-level nor purely sequence-level optimization is sufficient for complex reasoning. A hybrid approach that dynamically balances these granularities is key.
- Entropy as a Guide: Using token entropy to guide the mixing weight is a principled way to adapt the learning strategy. When the model is uncertain, it needs more fine-grained feedback (token-level); when confident, global consistency (sequence-level) is more beneficial.
- Branch-Specific Clipping is Crucial: Clipping independently before mixing is vital. It prevents the instability of one ratio type (e.g., high-variance token ratios) from corrupting the entire update, leading to more robust and stable learning.
- Exploration-Accuracy Link: The paper demonstrates a clear chain: sustained policy entropy (enabled by DHPO) leads to longer, more diverse trajectories, which in turn supports longer reasoning chains and ultimately higher accuracy.
Key Results
- Consistent Outperformance: DHPO consistently outperforms GRPO and GSPO across all tested benchmarks and model sizes (Qwen3-1.7B, 4B, 30B-A3B).
- Significant Accuracy Gains: For example, on Qwen3-30B-A3B-Base, DHPO-E (entropy-guided) achieved 34.4% accuracy on AIME24 (vs. 22.5% for GRPO) and 26.5% on AIME25 (vs. 14.6% for GRPO). On average, DHPO improved accuracy by ~4.9% over GRPO and ~4.3% over GSPO.
- Healthier Training Dynamics: DHPO maintains higher policy entropy and longer response lengths throughout training compared to GRPO and GSPO, indicating sustained exploration and avoiding premature collapse.
- Smoother Progress: The branch-specific clipping strategy leads to less oscillatory updates and more stable training dynamics, even when final performance is similar to simpler methods.
Limitations and Future Work
- Limited Model Family: The evaluation was restricted to the Qwen3 model family. The generality of DHPO across different architectures, tokenization schemes, or pretraining recipes remains to be explored.
- Focused Baseline Comparison: The study compared DHPO against a focused set of representative RLVR baselines, not an exhaustive evaluation of all existing methods.
- Future Work: The authors suggest further investigation into how hybrid importance mixing interacts with other variance-reduction or clipping strategies. Exploring DHPO's applicability to a wider range of LLM tasks beyond mathematical reasoning would also be valuable.
Summary updated