RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization

Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Yingwei Ma, Jue Chen, Binhua Li, Zhi Jin, Fei Huang, Yongbin Li, Ge Li

TL;DR

While Reinforcement Learning with Verifiable Reward (RLVR) significantly boosts the immediate accuracy of Large Language Models on reasoning tasks, it often causes a "capability boundary collapse" where the model's broader problem-solving scope shrinks. To fix this, the authors introduce RL-PLUS, a hybrid-policy optimization framework that safely integrates external data using Multiple Importance Sampling and incentivizes the exploration of low-probability, novel reasoning paths. RL-PLUS achieves state-of-the-art performance across multiple math and coding benchmarks while successfully expanding, rather than contracting, the model's capability boundaries.

What Problem Does This Paper Address?

Reinforcement Learning with Verifiable Reward (RLVR)—the paradigm behind models like DeepSeek-R1 and OpenAI's o1—uses rule-based rewards (like compiler outputs or math ground truths) to train LLMs to generate long, self-correcting chains of thought.

However, because these methods are typically on-policy and operate within massive action spaces with sparse rewards, they suffer from inward exploitation. The model becomes hyper-focused on optimizing the reasoning patterns it already knows. While this boosts its average first-try accuracy (pass@1), it causes capability boundary collapse. As shown in the pass@k metrics (the probability of finding a correct answer within $k$ attempts), standard RLVR models can actually perform worse than their un-trained base models when given multiple attempts (e.g., pass@128). The model's exploratory range shrinks, preventing it from discovering genuinely new reasoning pathways.

Figure 1: (a) The commonly used RLVR methods can lead to the collapse problem of capability boundaries in base LLMs. (b) RL-PLUS can overcome capability boundary collapse of LLMs in RLVR, consistently showing larger pass@k than base model.

Methods

RL-PLUS tackles this by introducing a hybrid-policy optimization approach that blends internal reinforcement learning with high-quality external data (off-policy trajectories). It relies on two core innovations:

  1. Multiple Importance Sampling (MIS): Standard off-policy learning introduces massive variance and bias when the model's current policy drifts away from the external data's distribution. RL-PLUS uses a mixture of the old policy and a uniform policy to construct a robust "variance guardrail," allowing the model to safely ingest external data without training instability.
  2. Exploration-Based Advantage Function: Inspired by focal loss, this component reshapes the reinforcement learning objective. It dynamically amplifies the advantage signal for reasoning paths that are correct but currently assigned a low probability by the model, forcing the LLM to explore unfamiliar territory.

Additionally, the authors intentionally remove the standard policy-gradient clipping mechanism. This empowers the model to take larger, more assertive optimization steps when it encounters highly informative, low-probability "new knowledge" in the external data.

Datasets

  • Training: A subset of 45,000 prompts from the OpenR1-Math-220k dataset, featuring correct reasoning trajectories annotated by DeepSeek-R1.
  • In-Domain Evaluation (Math): Six prominent mathematical reasoning benchmarks: AIME 24, AIME 25, AMC, MATH-500, Minerva, and Olympiad.
  • Out-of-Distribution Evaluation (OOD): Six tasks spanning science and coding to test generalization: ARC-c, GPQA-diamond, MMLU-Pro, HumanEval, LeetCode, and LiveCodeBench.

Key Insights and Ideas

  • The pass@k Illusion: Evaluating reasoning models purely on pass@1 accuracy masks a critical flaw. Standard RLVR acts like a student who crams a specific formula to pass an exam but loses the ability to think laterally, leading to a drop in pass@128 compared to the base model.
  • Preserving Policy Entropy: Directly injecting external data usually causes an "entropy explosion" (chaotic outputs), while standard RLVR causes "entropy collapse" (overly deterministic behavior). RL-PLUS maintains a healthy, non-zero policy entropy throughout training, meaning the model retains its capacity for creative exploration.
  • Bayesian Safety Nets: Because the true behavior policy behind external data is often unknown, RL-PLUS uses a Bayes-optimal estimator that mixes the proxy policy with a uniform distribution. This mathematical hedge prevents importance weights from exploding.

Key Results

  • State-of-the-Art Reasoning: Built on top of a Qwen2.5-Math-7B base, RL-PLUS achieved an average score of 53.4 across the six math benchmarks, outperforming standard GRPO (45.5) and SFT+GRPO (48.2).
  • Sustained Boundary Expansion: Unlike standard GRPO, whose performance advantage vanishes or dips below the base model as the number of samples ($k$) increases, RL-PLUS maintains a consistent edge all the way up to $k=128$.
  • Robust Generalization: RL-PLUS demonstrated strong out-of-distribution performance, scoring an average of 48.8 on OOD tasks. While standard SFT and GRPO methods degraded significantly under domain shifts like programming, RL-PLUS successfully transferred its reasoning capabilities.
Figure 2: Training dynamics of RL-PLUS and other baselines.

Limitations and Future Work

While RL-PLUS demonstrates excellent training stability and robustness, the authors note a few areas for further exploration:

  • Hyperparameter Tuning: Although the model consistently outperforms baselines across various values of the exploration hyperparameter $\gamma$, there is still room for improvement through fine-grained tuning.
  • Data Dependency: The framework relies on high-quality external reasoning trajectories (such as those generated by larger models like DeepSeek-R1) to effectively guide its outward exploration.

Summary updated

Abstract

Reinforcement Learning with Verifiable Reward (RLVR) has significantly advanced the complex reasoning abilities of Large Language Models (LLMs). However, it struggles to break through the inherent capability boundaries of the base LLM, due to its essentially on-policy strategy coupled with LLM's immense action space and sparse reward. Critically, RLVR can lead to the capability boundary collapse, narrowing the LLM's problem-solving scope. To address this problem, we propose RL-PLUS, a novel hybrid-policy optimization approach for LLMs that synergizes internal exploitation with external data to achieve stronger reasoning capabilities and surpass the boundaries of base models. RL-PLUS integrates two core components, i.e., Multiple Importance Sampling to address distributional mismatch from external data, and Exploration-Based Advantage Function to guide the model towards high-value, unexplored reasoning paths. We provide both theoretical analysis and extensive experiments to demonstrate the superiority and generalizability of our approach. Compared with existing RLVR methods, RL-PLUS achieves 1) state-of-the-art performance on six math reasoning benchmarks; 2) superior performance on six out-of-distribution reasoning tasks; 3) consistent and significant gains across diverse model families, with average relative improvements up to 69.2\%. Moreover, the analysis of Pass@k curves indicates that RL-PLUS effectively resolves the capability boundary collapse problem.
BibTeX
@inproceedings{dong2026rl,
  author = {Yihong Dong and Xue Jiang and Yongding Tao and Huanyu Liu and Kechi Zhang and Lili Mou and Rongyu Cao and Yingwei Ma and Jue Chen and Binhua Li and Zhi Jin and Fei Huang and Yongbin Li and Ge Li},
  title = {{RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1994},
}
Figure