Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration

Zhicheng Yang, Zhijiang Guo, Yinya Huang, Yongxin Wang, Dongchun Xie, Yiwei Wang, Xiaodan Liang, Jing Tang

Abstract

Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful paradigm for unlocking reasoning capabilities in large language models, yet its full potential is hindered by two under-explored dimensions: Depth—the hardest problem a model can sample; and Breadth—the number of instances consumed in a single iteration. We dissect the popular GRPO algorithm and reveal a systematic bias: the cumulative-advantage disproportionately weights samples with medium accuracy, while down-weighting the low-accuracy instances that are crucial for pushing reasoning boundaries. To rectify the depth neglect, we introduce Difficulty Adaptive Rollout Sampling (DARS), which re-weights hard problems through targeted multi-stage rollouts. Empirically, naively enlarging rollout size only accelerates convergence and even hurts Pass@K. Our DARS, in contrast, delivers consistent Pass@K gains without extra inference cost at convergence. Just as we adaptively expanded the depth of exploration, we now ask whether aggressively scaling the breadth of training data can further amplify reasoning gains. Increasing breadth significantly enhances Pass@1 performance by sustaining high token-level entropy and reducing gradient noise. We demonstrate that breadth and adaptive exploration across depth operate as orthogonal dimensions in RLVR, which are key to unleashing the reasoning power of LLMs.

The Challenge: Cumulative Advantage Bias in GRPO

Reinforcement Learning with Verifiable Reward (RLVR)—where models are rewarded for correct answers in math or code that can be checked automatically—is the engine behind recent reasoning breakthroughs. However, existing methods like Group Relative Policy Optimization (GRPO) suffer from cumulative advantage bias.

Because rewards are calculated relative to other samples in a small group, the algorithm disproportionately focuses on "medium-difficulty" problems where the model achieves a mix of success and failure. For "high-difficulty" problems where the model fails all initial attempts, the gradient vanishes, meaning the model never learns to solve the hardest tasks.

Figure 1: (a) Visualization of how standard GRPO underestimates hard samples by focusing on medium-accuracy regions. (b) The conceptual mechanism of DARS rebalancing rollouts to restore focus on hard problems.

Methodology: Depth and Breadth Synergy

To unlock the full potential of RLVR, we propose a two-pronged strategy addressing both the depth of exploration and the breadth of training.

Difficulty-Adaptive Rollout Sampling (DARS)

DARS addresses the "Depth" dimension using a two-phase approach. First, it performs a lightweight "pre-rollout" to estimate a problem's difficulty. Second, it dynamically allocates additional sampling budget to low-accuracy (hard) problems. We introduce two re-balancing schedules: Equal-Treatment (ET), which brings the advantage of hard problems up to the level of medium ones, and Hardness-Weighted (HW), which actively weights harder problems more heavily to force the model to focus on complex reasoning.

Breadth Scaling as Regularization

To address "Breadth," we replace standard PPO-style minibatch updates with full-batch updates across multiple epochs. Drastically increasing the batch size (e.g., from 128 to 3072) acts as an implicit entropy regularizer. This prevents the model from converging prematurely and significantly boosts Pass@1 metrics by sustaining higher token-level entropy throughout the training process.

Figure 5: Large-breadth training (higher batch sizes) sustains higher token-level entropy, preventing model collapse and improving Pass@1 performance.

Key Results

The combined DARS-HW-Breadth approach consistently outperformed baselines across different model scales (Qwen2.5-Math 1.5B/7B and Llama-3.1-8B):

  • Llama-3.1-8B: Achieved a significant jump in Pass@128, moving from a baseline of 56.5% to 68.7%.
  • Qwen2.5-Math-7B: Improved average Pass@1 while simultaneously raising the Pass@128 ceiling.
  • Efficiency: DARS achieved these gains while using significantly fewer total rollouts than naive scaling approaches, which often hurt Pass@K performance.
  • Reasoning Quality: Qualitative analysis showed that DARS-trained models produce longer and more structured reasoning traces, indicating they are learning to "think deeper" to solve hard problems.

Conclusion

Our findings confirm that "Depth" (adaptive exploration) and "Breadth" (scaling data quantity) are orthogonal dimensions. While DARS improves the model's potential (Pass@K), Breadth improves its consistency (Pass@1). Integrating these dimensions provides a robust framework for training LLMs on complex mathematical reasoning benchmarks like MATH-500 and OlympiadBench.

Summary updated

Abstract

Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful paradigm for unlocking reasoning capabilities in large language models, yet its full potential is hindered by two under-explored dimensions: Depth-the hardest problem a model can sample; Breadth-the number of instances consumed in a single iteration. We dissect the popular GRPO algorithm and reveal a systematic bias: the cumulative-advantage disproportionately weights samples with medium accuracy, while down-weighting the low-accuracy instances that are crucial for pushing reasoning boundaries. To rectify the depth neglect, we introduce Difficulty Adaptive Rollout Sampling (DARS), which re-weights hard problems through targeted multi-stage rollouts, thereby increasing the number of positive rollouts for hard problems. Empirically, naively enlarging rollout size only accelerates convergence and even hurts Pass@K. Our DARS, in contrast, delivers consistent Pass@K gains without extra inference cost at convergence. Just as we adaptively expanded the depth of exploration, we now ask whether aggressively scaling the breadth of training data can further amplify reasoning gains. To this end, we intensely scale batch size and replace PPO's mini-batch iterations with full-batch updates over multiple epochs. Increasing breadth significantly enhances Pass@1 performance. Large-breadth training sustains high token-level entropy, indicating continued exploration and reduced gradient noise. We further present DARS-B, which augments DARS with large breadth, and demonstrate simultaneous gains in Pass@K and Pass@1. The results confirm that breadth and adaptive exploration across depth operate as orthogonal dimensions in RLVR, which are key to unleashing the reasoning power of RLVR.
BibTeX
@inproceedings{yang2026depth,
  author = {Zhicheng Yang and Zhijiang Guo and Yinya Huang and Yongxin Wang and Dongchun Xie and Yiwei Wang and Xiaodan Liang and Jing Tang},
  title = {{Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=v3diR6NstK},
}
Figure