The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
Lists
Only you see this
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key method for improving Large Language Models' reasoning capabilities, yet recent evidence suggests it may paradoxically shrink the reasoning boundary rather than expand it. This paper investigates this shrinkage issue of RLVR by analyzing its learning dynamics and reveals two critical phenomena that explain this failure. First, we expose the negative interference phenomenon in RLVR, where learning to solve certain training problems actively reduces the likelihood of correct solutions for others, leading to the decline of Pass@$k$ performance or the probability of generating a correct solution within $k$ attempts. Second, we uncover the winner-take-all effect: RLVR disproportionately reinforces problems with high likelihood under the base model while suppressing initially low-likelihood, correct solutions. Through extensive theoretical and empirical analysis on multiple mathematical reasoning benchmarks, we show that this effect arises from the inherent on-policy sampling in standard RL objectives, causing the model to converge toward narrow solution strategies. Based on these insights, we propose a simple yet effective data curation algorithm that focuses RLVR learning on low-likelihood problems, achieving notable improvement in Pass@$k$ performance.
BibTeX
@misc{phuc2025reasoning,
author = {Nguyen Minh Phuc and Chinh Duc La and Duy Minh Ho Nguyen and Nitesh V Chawla and Binh Nguyen and Khoa D Doan},
title = {{The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models}},
year = {2025},
eprint = {2510.02230},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2510.02230},
}