TL;DR
While Reinforcement Learning with Verifiable Reward (RLVR) significantly boosts the immediate accuracy of Large Language Models on reasoning tasks, it often causes a "capability boundary collapse" where the model's broader problem-solving scope shrinks. To fix this, the authors introduce RL-PLUS, a hybrid-policy optimization framework that safely integrates external data using Multiple Importance Sampling and incentivizes the exploration of low-probability, novel reasoning paths. RL-PLUS achieves state-of-the-art performance across multiple math and coding benchmarks while successfully expanding, rather than contracting, the model's capability boundaries.
What Problem Does This Paper Address?
Reinforcement Learning with Verifiable Reward (RLVR)—the paradigm behind models like DeepSeek-R1 and OpenAI's o1—uses rule-based rewards (like compiler outputs or math ground truths) to train LLMs to generate long, self-correcting chains of thought.
However, because these methods are typically on-policy and operate within massive action spaces with sparse rewards, they suffer from inward exploitation. The model becomes hyper-focused on optimizing the reasoning patterns it already knows. While this boosts its average first-try accuracy (pass@1), it causes capability boundary collapse. As shown in the pass@k metrics (the probability of finding a correct answer within $k$ attempts), standard RLVR models can actually perform worse than their un-trained base models when given multiple attempts (e.g., pass@128). The model's exploratory range shrinks, preventing it from discovering genuinely new reasoning pathways.
Methods
RL-PLUS tackles this by introducing a hybrid-policy optimization approach that blends internal reinforcement learning with high-quality external data (off-policy trajectories). It relies on two core innovations:
- Multiple Importance Sampling (MIS): Standard off-policy learning introduces massive variance and bias when the model's current policy drifts away from the external data's distribution. RL-PLUS uses a mixture of the old policy and a uniform policy to construct a robust "variance guardrail," allowing the model to safely ingest external data without training instability.
- Exploration-Based Advantage Function: Inspired by focal loss, this component reshapes the reinforcement learning objective. It dynamically amplifies the advantage signal for reasoning paths that are correct but currently assigned a low probability by the model, forcing the LLM to explore unfamiliar territory.
Additionally, the authors intentionally remove the standard policy-gradient clipping mechanism. This empowers the model to take larger, more assertive optimization steps when it encounters highly informative, low-probability "new knowledge" in the external data.
Datasets
- Training: A subset of 45,000 prompts from the OpenR1-Math-220k dataset, featuring correct reasoning trajectories annotated by DeepSeek-R1.
- In-Domain Evaluation (Math): Six prominent mathematical reasoning benchmarks: AIME 24, AIME 25, AMC, MATH-500, Minerva, and Olympiad.
- Out-of-Distribution Evaluation (OOD): Six tasks spanning science and coding to test generalization: ARC-c, GPQA-diamond, MMLU-Pro, HumanEval, LeetCode, and LiveCodeBench.
Key Insights and Ideas
- The
pass@kIllusion: Evaluating reasoning models purely onpass@1accuracy masks a critical flaw. Standard RLVR acts like a student who crams a specific formula to pass an exam but loses the ability to think laterally, leading to a drop inpass@128compared to the base model. - Preserving Policy Entropy: Directly injecting external data usually causes an "entropy explosion" (chaotic outputs), while standard RLVR causes "entropy collapse" (overly deterministic behavior). RL-PLUS maintains a healthy, non-zero policy entropy throughout training, meaning the model retains its capacity for creative exploration.
- Bayesian Safety Nets: Because the true behavior policy behind external data is often unknown, RL-PLUS uses a Bayes-optimal estimator that mixes the proxy policy with a uniform distribution. This mathematical hedge prevents importance weights from exploding.
Key Results
- State-of-the-Art Reasoning: Built on top of a Qwen2.5-Math-7B base, RL-PLUS achieved an average score of 53.4 across the six math benchmarks, outperforming standard GRPO (45.5) and SFT+GRPO (48.2).
- Sustained Boundary Expansion: Unlike standard GRPO, whose performance advantage vanishes or dips below the base model as the number of samples ($k$) increases, RL-PLUS maintains a consistent edge all the way up to $k=128$.
- Robust Generalization: RL-PLUS demonstrated strong out-of-distribution performance, scoring an average of 48.8 on OOD tasks. While standard SFT and GRPO methods degraded significantly under domain shifts like programming, RL-PLUS successfully transferred its reasoning capabilities.
Limitations and Future Work
While RL-PLUS demonstrates excellent training stability and robustness, the authors note a few areas for further exploration:
- Hyperparameter Tuning: Although the model consistently outperforms baselines across various values of the exploration hyperparameter $\gamma$, there is still room for improvement through fine-grained tuning.
- Data Dependency: The framework relies on high-quality external reasoning trajectories (such as those generated by larger models like DeepSeek-R1) to effectively guide its outward exploration.
Summary updated