Abstract
Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful paradigm for unlocking reasoning capabilities in large language models, yet its full potential is hindered by two under-explored dimensions: Depth—the hardest problem a model can sample; and Breadth—the number of instances consumed in a single iteration. We dissect the popular GRPO algorithm and reveal a systematic bias: the cumulative-advantage disproportionately weights samples with medium accuracy, while down-weighting the low-accuracy instances that are crucial for pushing reasoning boundaries. To rectify the depth neglect, we introduce Difficulty Adaptive Rollout Sampling (DARS), which re-weights hard problems through targeted multi-stage rollouts. Empirically, naively enlarging rollout size only accelerates convergence and even hurts Pass@K. Our DARS, in contrast, delivers consistent Pass@K gains without extra inference cost at convergence. Just as we adaptively expanded the depth of exploration, we now ask whether aggressively scaling the breadth of training data can further amplify reasoning gains. Increasing breadth significantly enhances Pass@1 performance by sustaining high token-level entropy and reducing gradient noise. We demonstrate that breadth and adaptive exploration across depth operate as orthogonal dimensions in RLVR, which are key to unleashing the reasoning power of LLMs.
The Challenge: Cumulative Advantage Bias in GRPO
Reinforcement Learning with Verifiable Reward (RLVR)—where models are rewarded for correct answers in math or code that can be checked automatically—is the engine behind recent reasoning breakthroughs. However, existing methods like Group Relative Policy Optimization (GRPO) suffer from cumulative advantage bias.
Because rewards are calculated relative to other samples in a small group, the algorithm disproportionately focuses on "medium-difficulty" problems where the model achieves a mix of success and failure. For "high-difficulty" problems where the model fails all initial attempts, the gradient vanishes, meaning the model never learns to solve the hardest tasks.
Methodology: Depth and Breadth Synergy
To unlock the full potential of RLVR, we propose a two-pronged strategy addressing both the depth of exploration and the breadth of training.
Difficulty-Adaptive Rollout Sampling (DARS)
DARS addresses the "Depth" dimension using a two-phase approach. First, it performs a lightweight "pre-rollout" to estimate a problem's difficulty. Second, it dynamically allocates additional sampling budget to low-accuracy (hard) problems. We introduce two re-balancing schedules: Equal-Treatment (ET), which brings the advantage of hard problems up to the level of medium ones, and Hardness-Weighted (HW), which actively weights harder problems more heavily to force the model to focus on complex reasoning.
Breadth Scaling as Regularization
To address "Breadth," we replace standard PPO-style minibatch updates with full-batch updates across multiple epochs. Drastically increasing the batch size (e.g., from 128 to 3072) acts as an implicit entropy regularizer. This prevents the model from converging prematurely and significantly boosts Pass@1 metrics by sustaining higher token-level entropy throughout the training process.
Key Results
The combined DARS-HW-Breadth approach consistently outperformed baselines across different model scales (Qwen2.5-Math 1.5B/7B and Llama-3.1-8B):
- Llama-3.1-8B: Achieved a significant jump in Pass@128, moving from a baseline of 56.5% to 68.7%.
- Qwen2.5-Math-7B: Improved average Pass@1 while simultaneously raising the Pass@128 ceiling.
- Efficiency: DARS achieved these gains while using significantly fewer total rollouts than naive scaling approaches, which often hurt Pass@K performance.
- Reasoning Quality: Qualitative analysis showed that DARS-trained models produce longer and more structured reasoning traces, indicating they are learning to "think deeper" to solve hard problems.
Conclusion
Our findings confirm that "Depth" (adaptive exploration) and "Breadth" (scaling data quantity) are orthogonal dimensions. While DARS improves the model's potential (Pass@K), Breadth improves its consistency (Pass@1). Integrating these dimensions provides a robust framework for training LLMs on complex mathematical reasoning benchmarks like MATH-500 and OlympiadBench.
Summary updated