When The Verifier Is The Only Trustworthy Feedback Source: A Self-Teacher RLVR Pilot, A Confounded Logprob Extension, And Four Corrective Probes

RLxF @ ICML 2026 · OpenReview

TL;DR

Researchers investigated whether Large Language Models (LLMs) can judge their own reasoning well enough to bootstrap their way to solving harder math problems. While initial tests suggested models were blind to their own errors, the authors discovered a "measurement confound": models were simply copying visible answers and ignoring the logic. By using new "probes"—like asking the model for a YES/NO verdict or measuring its "surprise" at specific tokens—they proved that models do have an internal "BS detector" that is significantly amplified by instruction tuning.

What Problem Does This Paper Address?

In the quest to build smarter AI, a popular idea is Self-Teacher Reinforcement Learning from Verifiable Rewards (RLVR). The premise is based on the "Generative AI Paradox": models are often better at recognizing a correct solution than they are at generating one from scratch. If a model can act as its own critic, we could theoretically use that internal judgment to "bootstrap" its performance on hard problems without needing a smarter "teacher" model (like GPT-4) to provide labels.

However, when the authors ran a pre-registered behavioral test on the Qwen2.5-1.5B model using hard competition math, the model failed to show any benefit from seeing correct reasoning chains versus corrupted ones. This paper investigates why that signal appeared to be missing and how to actually find it.

Methods

The authors moved beyond simple performance metrics to look at the model's internal probability distributions (logprobs). They identified a fatal flaw in standard testing: if a reasoning chain ends with a visible, correct answer, the model will assign high probability to that answer regardless of whether the steps leading up to it are logical or complete gibberish. This is called answer-copying.

To bypass this, they developed four "corrective probes":

  1. Answer-Relabeled Corruption: They changed an intermediate step and the final answer by the same amount, making the chain self-consistent but wrong. This forces the model to actually "read" the chain to predict the matching (wrong) answer.
  2. Verdict YES/NO: Instead of predicting the answer, the model is prompted to act as a grader and output "YES" or "NO" regarding the chain's correctness.
  3. Prompt-Format Ablation: Testing if the way the problem is framed (e.g., chat vs. bare text) changes the result.
  4. Per-Token Surprise: Measuring the negative log-likelihood at every single token to see if the model's "surprise" spikes exactly at the point where a reasoning error is introduced.

Datasets

  • AIME (American Invitational Mathematics Examination): Problems from 2018–2023. The authors specifically curated a "hard set" of 51 problems that the Qwen2.5-1.5B base model could never solve in 1,024 attempts.
  • NuminaMath: Used to provide "Oracle" (known-correct) step-by-step solutions for the problems.
  • Synthetic Control: 50 fresh, SymPy-verified synthetic problems to ensure the model wasn't just relying on memorized training data.

Key Insights and Ideas

  • The "Measurement-Gating" Trap: You can't measure a model's reasoning sensitivity if you give it a "shortcut" (like a visible final answer). Most previous failures to find a self-correction signal might have been artifacts of the testing protocol, not the model's intelligence.
  • Instruction Tuning is a Discrimination Multiplier: One of the most striking findings was that instruction tuning (the process of making a model follow directions) doesn't just make it more helpful—it makes it a much better critic. At the 7B parameter scale, instruction tuning amplified the model's ability to distinguish correct from incorrect chains by ~5x.
  • Token-Level Resolution: The "Per-Token Surprise" probe showed that models are locally sensitive to errors. When a number is changed in a math step, the model's surprise spikes at that exact token. This provides a "dense" feedback signal that external verifiers (which only check the final answer) cannot provide.

Key Results

  • Initial Failure: In the behavioral test, the 1.5B model produced the correct answer at a rate of 1.23% regardless of whether it saw a correct or corrupted chain.
  • Probe Success: Once the "answer-copying" confound was removed (Probe 1), 100% of tested models showed sensitivity to chain content.
  • Verdict Discrimination: The "YES/NO" verdict gap (Probe 2) ranged from a tiny 0.15 logprob units in distilled models to a massive 7.34 units in Qwen-7B-Instruct.
  • The "R1-Zero" Pattern: The authors noted that DeepSeek's R1-Distill models actually had the weakest internal discrimination, suggesting that those models rely almost entirely on the external verifier rather than internal judgment.

Limitations and Future Work

  • Scope: The study is limited to competition math (AIME) and the Qwen model family. It remains to be seen if these findings generalize to code generation or formal theorem proving.
  • The "Spontaneous Generation" Bottleneck: Even though the models are good critics, the behavioral test still failed. This means that having a "BS detector" doesn't automatically mean the model can suddenly generate the correct chain on its own.
  • Future Direction: The authors suggest that the next step is to use these "clean" signals (verdicts and surprise) to actually train the models, rather than just measuring them. They recommend starting RL from an Instruct checkpoint rather than a Base checkpoint to take advantage of the 5x stronger discrimination signal.

Summary updated

Abstract

Self-teacher reinforcement learning from verifiable rewards (RLVR) proposes that a base language model can judge its own reasoning chains well enough to bootstrap improvement without an external neural teacher — the model's own discrimination serves as a second feedback channel alongside the formal verifier. We pre-register a behavioral test of this premise on Qwen2.5-1.5B base over 51 hard competition-math problems where the model never produces a correct answer. The test fails on every sub-criterion. A 17-measurement log-probability extension across base, Instruct, and R1-distilled checkpoints (1.5B–72B) initially reproduces this null, but we then uncover a confound: the corruption protocol preserves the visible final answer, so the test cannot detect chain-content sensitivity even where it exists. Four corrective probes fix this and show that (1) every model is sensitive to chain content at every scale we tested, and (2) instruction tuning amplifies verdict-style discrimination by ~5x at 7B. For the RLxF practitioner: the external verifier (SymPy, Lean 4, code executor) remains the trustworthy world-feedback channel. The base's internal discrimination is real but measurement-gated — any logprob test that anchors on a visible final answer is confounded. The verdict YES/NO and per-token surprise probes recover clean signals. The pre-registered behavioral test still fails, so pipelines that need the base to spontaneously produce correct chains remain bottlenecked. But pipelines that read only the discrimination side of the loop are not refuted here.
BibTeX
@inproceedings{2026when,
  title = {{When The Verifier Is The Only Trustworthy Feedback Source: A Self-Teacher RLVR Pilot, A Confounded Logprob Extension, And Four Corrective Probes}},
  booktitle = {RLxF @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=VM4qPAZAV5},
}
Figure