Abstract
Large reasoning models (LRMs) often exhibit computational inefficiency, generating excessive tokens for simple tasks or suffering performance degradation when constrained by standard length penalties. We introduce Hierarchical Budget Policy Optimization (HBPO), a reinforcement learning framework that enables adaptive reasoning. By partitioning training into token-budget hierarchies and employing a piecewise reward function, HBPO allows models to scale their "thinking" to problem complexity. Our results demonstrate that HBPO reduces average token usage by over 60% while improving accuracy by up to 3.14% across diverse mathematical and scientific benchmarks.
Introduction: The Efficiency-Capability Trade-off
Current reasoning models like DeepSeek-R1 and OpenAI’s o1 utilize Chain-of-Thought (CoT) processing to achieve high accuracy. However, this deliberation is often inefficient, with models frequently over-calculating simple problems. Existing solutions typically involve length penalties or discrete "think/no-think" modes. Length penalties often lead to "exploration space collapse," where the model becomes overly concise and loses its reasoning depth. Discrete modes are too binary to handle the nuanced spectrum of real-world problem difficulty.
HBPO addresses these issues by treating reasoning efficiency as a learned capability rather than a fixed constraint, aiming for "adaptive reasoning"—using minimal tokens for simple tasks and extensive deliberation for complex ones.
Methods: Hierarchical Budget Policy Optimization
HBPO builds upon the Group Relative Policy Optimization (GRPO) framework, introducing three core components:
1. Hierarchical Budgeting
The model is trained to solve queries under varying token constraints (e.g., 512, 1024, 2048, and 2560 tokens). This multi-level approach ensures the model maintains the ability to be thorough while practicing conciseness.
2. Piecewise Reward Function
To avoid the pitfalls of linear penalties, HBPO utilizes a piecewise structure. Within the assigned budget, rewards are monotonic to encourage flexibility. Beyond the budget, a combination of cosine decay and deviation penalties guides the policy back toward the target length without causing the model to "give up" on complex reasoning.
3. Dual-Level Advantages
The training signal is split into intra-subgroup advantages (comparing performance within a specific budget bucket) and inter-subgroup advantages (comparing against the global average). This teaches the model to identify the most appropriate budget for a given problem complexity.
Experimental Results
We evaluated HBPO using the DeepScaleR dataset (40,000 math problems) on benchmarks including GSM8K, Math500, AIME25, and GPQA-Diamond.
Efficiency and Accuracy Gains
HBPO achieved a 60.6% reduction in average token usage (from 7,921 to 3,120 tokens) on the DeepSeek-R1-Distill-Qwen-1.5B base model. Crucially, this efficiency did not come at the cost of accuracy; instead, average accuracy improved by 3.14%. On the challenging AIME25 benchmark, HBPO reached 31.1% accuracy, significantly outperforming the baseline.
Emergent Adaptivity
A key finding is the model's ability to self-correct and scale. On simple GSM8K problems, the model naturally used ~670 tokens, while on AIME25, it automatically scaled up to ~5,600 tokens. This indicates that the model learned the fundamental relationship between problem complexity and required compute.
Discussion and Limitations
HBPO demonstrates that efficiency is a capability that enhances accuracy by reducing redundant reasoning and potential hallucinations. However, limitations remain:
- Context Window: Training was restricted to 4,096 tokens, which may limit extremely long-form reasoning.
- Fixed Hierarchies: Budget levels were manually selected; future iterations could explore dynamic budget discovery.
- Token Counting: The model's internal sense of token length is imperfect, suggesting a need for more robust internal tracking mechanisms.
Conclusion
Hierarchical Budget Policy Optimization provides a scalable path toward efficient, adaptive reasoning in large language models. By preserving the "long tail" of deliberation while incentivizing concisenessness, HBPO ensures that computational resources are allocated where they are most needed.
Summary updated