RL Can Only Reinforce What It Sees Succeed
Lists
Only you see this
Abstract
When a familiar rule stops working, an agent has to weigh what pre-training taught it against what it now observes. We ask how outcome-reward reinforcement learning (RL) changes this balance, in a puzzle world where both sides are known: small language models trained from scratch on games played by a Bayesian player, so that the prior and the right answer are exact. Base models hedge as their corpus did, and RL turns that hedging into decisions, in whichever direction the reward favours. But RL can only reinforce what it sees succeed. When the right action is rare, each run either learns it everywhere or not at all, and which runs take off is predicted before training by one count: the expected number of groups of attempts in which the action wins. About 35 such groups give even odds. The count holds for new and repeated problems and for groups of 4 to 256 attempts, and across learning rates, optimizers, GRPO variants and model sizes from 1.3M to 85M parameters, which move the threshold but not what is counted.
BibTeX
@misc{sanchezcorcuera2026rl,
author = {Ruben Sanchez-Corcuera and Unai Zulaika and Aritz Bilbao-Jayo and Miguel Fernandez-de-Retana},
title = {{RL Can Only Reinforce What It Sees Succeed}},
year = {2026},
eprint = {2611.26084},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.26084},
}Loading related content...