TL;DR
Reinforcement Learning with Verifiable Rewards (RLVR)—the technique behind models like DeepSeek-R1—doesn't actually teach models how to reason better. Instead, it "sharpens" the model's output, making it more likely to select the correct reasoning paths it already possessed. By tracking internal states across training, researchers found that while performance improves, the internal representations of reasoning errors remain completely unchanged.
What Problem Does This Paper Address?
The AI community has been buzzing about RLVR, where models are rewarded for correct answers (like math results) without being told how to get there. This process leads to dramatic jumps in reasoning benchmarks. The prevailing theory is "sharpening": the base model already "knows" the logic, and RL training simply narrows the output distribution to favor those successful paths.
However, most evidence for this has been behavioral (looking at what the model outputs) or based on "before and after" snapshots. This paper addresses a critical gap: Does the internal "mind" of the model actually change during training? Specifically, do the internal representations of reasoning errors evolve as the model gets "smarter" through RL?
Methods
The authors performed a longitudinal study of OLMo-2-1B across 13 training checkpoints on grade-school math. They used two primary lenses:
- Behavioral Sweeps: They measured
pass@k(the probability of getting a correct answer in k attempts). If the "sharpening" hypothesis is true, the model's total "coverage" (the set of problems it can solve given enough tries) should stay the same while its first-try accuracy (pass@1) improves. - Representational Probing: To see inside the "mind," they deliberately planted three types of arithmetic errors (e.g., swapping an operator or scaling a result) into correctly solved problems. They then trained simple linear probes (logistic regression) on the model's internal hidden states to see if the model could "detect" these errors.
- Positive Control: They also trained probes to see if internal activations could distinguish between early and late training checkpoints. This served as a "sanity check" to ensure their methods could detect change if it actually existed.
Datasets
The study focused on GSM8K (Grade School Math 8K), specifically the test split of 1,319 problems. This dataset is ideal because it provides unambiguous ground truth for what constitutes a reasoning error.
Key Insights and Ideas
- The Knowledge was Already There: At the very first checkpoint, the model's
pass@256(the chance of getting it right in 256 tries) was already 94.5%. RLVR didn't "teach" the model math; it just helped the model "find" the right answer on the first try more often. - Static Internal Logic: The probes' ability to detect reasoning errors (measured by AUROC) remained almost perfectly flat across all 13 checkpoints. The model's internal "signature" for a mistake didn't get sharper or more sophisticated; it stayed exactly the same.
- The "Final Answer" Footprint: The only place the researchers found significant representational change was at the final answer token. A probe could easily tell an early checkpoint from a late one at the very end of the string, but it couldn't tell them apart during the intermediate reasoning steps.
- RLVR as a Distribution Tweak: The results suggest RLVR acts like a filter on the output distribution rather than a fundamental rewiring of the model's logic. It "sharpens the pencil" (the output) without changing the "mind" (the internal reasoning process).
Key Results
- Accuracy Gains:
pass@1rose from 58.0% to 61.3% over training. Crucially,pass@256also rose slightly (94.5% to 96.9%), showing that the model's total solution coverage didn't shrink (a phenomenon called "mode collapse" often seen in larger models).
- Invariance of Errors: Probes detecting injected errors maintained an AUROC between 0.798 and 0.840 across all checkpoints. This spread is smaller than the statistical noise, meaning the internal representation of errors is invariant to RL training.
- Spatial Dissociation: The "Positive Control" probe hit an AUROC of 0.93 at the final answer token when trying to identify the checkpoint, but dropped to near-chance (0.5) during the actual reasoning steps. This proves that RLVR's "representational footprint" is concentrated at the very end of the chain.
Limitations and Future Work
The authors are careful to scope their claims:
- Single Model/Task: The study used one model (OLMo-2-1B) and one task (arithmetic). It remains to be seen if these findings hold for much larger models (like 70B+) or more abstract reasoning tasks where "verifiable rewards" are harder to define.
- Diagnostic vs. Causal: Linear probes show that information is present, but they don't prove the model uses that information to make decisions. However, the authors argue that their positive control (finding change at the output token) makes their "null result" during reasoning steps much more meaningful.
- Future Direction: A practical takeaway is that error-detecting verifiers trained on base models should transfer well to RL-tuned models, potentially saving significant compute in verification pipelines.
Summary updated