Hierarchical Budget Policy Optimization for Adaptive Reasoning

Shangke Lyu, Linjuan Wu, Yuchen Yan, Xingyu Wu, Hao Li, Yongliang Shen, Peisheng Jiang, Weiming Lu, Jun Xiao, Yueting ZhuangarXiv 2025 · DOI · arXiv

Abstract

Large reasoning models (LRMs) often exhibit computational inefficiency, generating excessive tokens for simple tasks or suffering performance degradation when constrained by standard length penalties. We introduce Hierarchical Budget Policy Optimization (HBPO), a reinforcement learning framework that enables adaptive reasoning. By partitioning training into token-budget hierarchies and employing a piecewise reward function, HBPO allows models to scale their "thinking" to problem complexity. Our results demonstrate that HBPO reduces average token usage by over 60% while improving accuracy by up to 3.14% across diverse mathematical and scientific benchmarks.

Introduction: The Efficiency-Capability Trade-off

Current reasoning models like DeepSeek-R1 and OpenAI’s o1 utilize Chain-of-Thought (CoT) processing to achieve high accuracy. However, this deliberation is often inefficient, with models frequently over-calculating simple problems. Existing solutions typically involve length penalties or discrete "think/no-think" modes. Length penalties often lead to "exploration space collapse," where the model becomes overly concise and loses its reasoning depth. Discrete modes are too binary to handle the nuanced spectrum of real-world problem difficulty.

Comparison of reasoning optimization methods: Length-constrained, Discrete mode, and the proposed Hierarchical Budget Policy Optimization.

HBPO addresses these issues by treating reasoning efficiency as a learned capability rather than a fixed constraint, aiming for "adaptive reasoning"—using minimal tokens for simple tasks and extensive deliberation for complex ones.

Methods: Hierarchical Budget Policy Optimization

HBPO builds upon the Group Relative Policy Optimization (GRPO) framework, introducing three core components:

1. Hierarchical Budgeting

The model is trained to solve queries under varying token constraints (e.g., 512, 1024, 2048, and 2560 tokens). This multi-level approach ensures the model maintains the ability to be thorough while practicing conciseness.

Detailed view of the Hierarchical Budget Policy Optimization mechanism showing budget-aware rewards across different token limits.

2. Piecewise Reward Function

To avoid the pitfalls of linear penalties, HBPO utilizes a piecewise structure. Within the assigned budget, rewards are monotonic to encourage flexibility. Beyond the budget, a combination of cosine decay and deviation penalties guides the policy back toward the target length without causing the model to "give up" on complex reasoning.

3. Dual-Level Advantages

The training signal is split into intra-subgroup advantages (comparing performance within a specific budget bucket) and inter-subgroup advantages (comparing against the global average). This teaches the model to identify the most appropriate budget for a given problem complexity.

Experimental Results

We evaluated HBPO using the DeepScaleR dataset (40,000 math problems) on benchmarks including GSM8K, Math500, AIME25, and GPQA-Diamond.

Efficiency and Accuracy Gains

HBPO achieved a 60.6% reduction in average token usage (from 7,921 to 3,120 tokens) on the DeepSeek-R1-Distill-Qwen-1.5B base model. Crucially, this efficiency did not come at the cost of accuracy; instead, average accuracy improved by 3.14%. On the challenging AIME25 benchmark, HBPO reached 31.1% accuracy, significantly outperforming the baseline.

Emergent Adaptivity

A key finding is the model's ability to self-correct and scale. On simple GSM8K problems, the model naturally used ~670 tokens, while on AIME25, it automatically scaled up to ~5,600 tokens. This indicates that the model learned the fundamental relationship between problem complexity and required compute.

Discussion and Limitations

HBPO demonstrates that efficiency is a capability that enhances accuracy by reducing redundant reasoning and potential hallucinations. However, limitations remain:

  • Context Window: Training was restricted to 4,096 tokens, which may limit extremely long-form reasoning.
  • Fixed Hierarchies: Budget levels were manually selected; future iterations could explore dynamic budget discovery.
  • Token Counting: The model's internal sense of token length is imperfect, suggesting a need for more robust internal tracking mechanisms.

Conclusion

Hierarchical Budget Policy Optimization provides a scalable path toward efficient, adaptive reasoning in large language models. By preserving the "long tail" of deliberation while incentivizing concisenessness, HBPO ensures that computational resources are allocated where they are most needed.

Summary updated

Abstract

Large reasoning models achieve remarkable performance through extensive chain-of-thought generation, yet they suffer from a critical inefficiency: applying uniformly extensive reasoning regardless of problem complexity. We present Hierarchical Budget Policy Optimization (HBPO), a reinforcement learning framework that enables models to learn problem-specific reasoning depths without sacrificing capability. Unlike existing approaches that impose rigid constraints or rely on discrete mode selection, HBPO partitions the exploration space into budget-constrained hierarchies (512-2560 tokens), each with differentiated reward structures that preserve both efficiency incentives and reasoning capabilities. This design addresses a fundamental challenge in efficient reasoning training: traditional length penalties systematically bias models away from necessary long reasoning paths, causing exploration space collapse. Through hierarchical sampling and budget-aware rewards, HBPO maintains exploration diversity while teaching models to recognize when extended deliberation is warranted. Extensive experiments demonstrate that HBPO reduces average token usage by up to 60.6% while improving accuracy by 3.14% across four reasoning benchmarks. Most notably, HBPO exhibits emergent adaptive behavior where models automatically adjust reasoning depth based on problem complexity. Our results suggest that reasoning efficiency and capability are not inherently conflicting, and can be simultaneously optimized through appropriately structured hierarchical training that preserves exploration diversity.
BibTeX
@misc{lyu2025hierarchical,
  author = {Shangke Lyu and Linjuan Wu and Yuchen Yan and Xingyu Wu and Hao Li and Yongliang Shen and Peisheng Jiang and Weiming Lu and Jun Xiao and Yueting Zhuang},
  title = {{Hierarchical Budget Policy Optimization for Adaptive Reasoning}},
  year = {2025},
  eprint = {2507.15844},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.15844},
}
Figure