RLVR Sharpens the Pencil, Not the Mind

Shreeyam Bangera, Manya GuptaarXiv 2026 · arXiv

TL;DR

Reinforcement Learning with Verifiable Rewards (RLVR)—the technique behind models like DeepSeek-R1—doesn't actually teach models how to reason better. Instead, it "sharpens" the model's output, making it more likely to select the correct reasoning paths it already possessed. By tracking internal states across training, researchers found that while performance improves, the internal representations of reasoning errors remain completely unchanged.

What Problem Does This Paper Address?

The AI community has been buzzing about RLVR, where models are rewarded for correct answers (like math results) without being told how to get there. This process leads to dramatic jumps in reasoning benchmarks. The prevailing theory is "sharpening": the base model already "knows" the logic, and RL training simply narrows the output distribution to favor those successful paths.

However, most evidence for this has been behavioral (looking at what the model outputs) or based on "before and after" snapshots. This paper addresses a critical gap: Does the internal "mind" of the model actually change during training? Specifically, do the internal representations of reasoning errors evolve as the model gets "smarter" through RL?

Methods

The authors performed a longitudinal study of OLMo-2-1B across 13 training checkpoints on grade-school math. They used two primary lenses:

  1. Behavioral Sweeps: They measured pass@k (the probability of getting a correct answer in k attempts). If the "sharpening" hypothesis is true, the model's total "coverage" (the set of problems it can solve given enough tries) should stay the same while its first-try accuracy (pass@1) improves.
  2. Representational Probing: To see inside the "mind," they deliberately planted three types of arithmetic errors (e.g., swapping an operator or scaling a result) into correctly solved problems. They then trained simple linear probes (logistic regression) on the model's internal hidden states to see if the model could "detect" these errors.
  3. Positive Control: They also trained probes to see if internal activations could distinguish between early and late training checkpoints. This served as a "sanity check" to ensure their methods could detect change if it actually existed.

Datasets

The study focused on GSM8K (Grade School Math 8K), specifically the test split of 1,319 problems. This dataset is ideal because it provides unambiguous ground truth for what constitutes a reasoning error.

Key Insights and Ideas

  • The Knowledge was Already There: At the very first checkpoint, the model's pass@256 (the chance of getting it right in 256 tries) was already 94.5%. RLVR didn't "teach" the model math; it just helped the model "find" the right answer on the first try more often.
  • Static Internal Logic: The probes' ability to detect reasoning errors (measured by AUROC) remained almost perfectly flat across all 13 checkpoints. The model's internal "signature" for a mistake didn't get sharper or more sophisticated; it stayed exactly the same.
  • The "Final Answer" Footprint: The only place the researchers found significant representational change was at the final answer token. A probe could easily tell an early checkpoint from a late one at the very end of the string, but it couldn't tell them apart during the intermediate reasoning steps.
  • RLVR as a Distribution Tweak: The results suggest RLVR acts like a filter on the output distribution rather than a fundamental rewiring of the model's logic. It "sharpens the pencil" (the output) without changing the "mind" (the internal reasoning process).

Key Results

  • Accuracy Gains: pass@1 rose from 58.0% to 61.3% over training. Crucially, pass@256 also rose slightly (94.5% to 96.9%), showing that the model's total solution coverage didn't shrink (a phenomenon called "mode collapse" often seen in larger models).
Figure 1: First-try accuracy and deep coverage across training. (a) pass@k curves from the deep sweep. (b) pass@1 rises by +3.3 pts while pass@256 does not narrow; the model’s solution coverage stays intact across training.
  • Invariance of Errors: Probes detecting injected errors maintained an AUROC between 0.798 and 0.840 across all checkpoints. This spread is smaller than the statistical noise, meaning the internal representation of errors is invariant to RL training.
  • Spatial Dissociation: The "Positive Control" probe hit an AUROC of 0.93 at the final answer token when trying to identify the checkpoint, but dropped to near-chance (0.5) during the actual reasoning steps. This proves that RLVR's "representational footprint" is concentrated at the very end of the chain.
Figure 2: Error-detection AUROC remains flat across training steps (a), while checkpoint identity is only detectable at the final answer (b).

Limitations and Future Work

The authors are careful to scope their claims:

  • Single Model/Task: The study used one model (OLMo-2-1B) and one task (arithmetic). It remains to be seen if these findings hold for much larger models (like 70B+) or more abstract reasoning tasks where "verifiable rewards" are harder to define.
  • Diagnostic vs. Causal: Linear probes show that information is present, but they don't prove the model uses that information to make decisions. However, the authors argue that their positive control (finding change at the output token) makes their "null result" during reasoning steps much more meaningful.
  • Future Direction: A practical takeaway is that error-detecting verifiers trained on base models should transfer well to RL-tuned models, potentially saving significant compute in verification pipelines.

Summary updated

Abstract

Models like DeepSeek-R1 made Reinforcement Learning with Verifiable Rewards (RLVR) famous: reward a model for correct answers and it gets dramatically better at reasoning without being taught how. The leading explanation is sharpening---that the base model already knows how to solve these problems and RLVR just makes it more likely to reach for its best solution paths. This has only ever been checked by comparing before and after training, never by watching what changes inside across it. We tracked OLMo-2-1B across all 13 training checkpoints on GSM8K. First-try accuracy climbs from 58.0% to 61.3%, but ask the model 256 times and solution coverage actually rises from 94.5% to 96.9%, proving the knowledge was already there. We then planted reasoning mistakes into problems and trained simple probes to detect them. Those probes held between AUROC 0.798 and 0.840 across all 13 checkpoints and 3 error types, a spread smaller than the confidence interval; the model never builds a sharper internal picture of what a reasoning mistake looks like. The one place we find real internal change is at the final answer token, where a probe distinguishing early from late checkpoints hits an AUROC of 0.93, while the same probe on the actual reasoning steps lands near chance. RLVR makes the model lead with its best thinking. It does not change what that thinking is.
BibTeX
@misc{bangera2026rlvr,
  author = {Shreeyam Bangera and Manya Gupta},
  title = {{RLVR Sharpens the Pencil, Not the Mind}},
  year = {2026},
  eprint = {2611.02574},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02574},
}
Figure