A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning

Mayug Maniparambil, Ayush Rai, Ekansh Verma, Noel E. O'Connor, Fergal ReidarXiv 2026 · arXiv

TL;DR

Standard self-distillation for LLM reasoning fails because simply scoring a trace with the answer in context doesn't actually pinpoint errors. This paper introduces Critique-Conditioned Self-Distillation, showing that forcing a model to first generate a critique creates a dense, corrective signal that successfully localizes mistakes and improves both test-time performance and reinforcement learning.

What Problem Does This Paper Address?

Reinforcement Learning with Verifiable Rewards (RLVR) is the gold standard for training reasoning models, but it suffers from "sparse" feedback—you only know if the final answer is right or wrong, not which specific step led to the failure. This is particularly problematic in "all-wrong groups," where every sampled attempt for a problem fails, leaving the model with no gradient to learn from.

To solve this, researchers use On-Policy Self-Distillation (OPSD). In this setup, a "teacher" (the model itself, but given the reference solution) scores the "student's" (the model without the answer) reasoning trace. The hope is that the teacher will provide "dense" feedback at every token. However, this paper discovers a structural flaw: a teacher that only scores a trace doesn't actually "search" for the error. Consequently, the feedback signal is often flat and fails to distinguish between correct and incorrect reasoning steps.

Methods

The authors investigate why the standard scoring signal is "empty" and propose a generative fix.

  1. Signal Analysis: They measure two primary signals:
  2. Localization (KL Divergence): Does the teacher's distribution change significantly at the exact moment an error occurs?
  3. Correction (Token Log-Ratio): Does the teacher actively push probability away from the student's erroneous tokens?
  4. Gold Error Spans: They use high-capability models (DeepSeek, Claude) to annotate exactly where reasoning traces go wrong, creating a "ground truth" for error localization.
  5. Critique-Conditioned Self-Distillation: Instead of just giving the teacher the reference solution, they ask the model to first generate a critique (a verbatim quote of the error plus an explanation). They then condition the scoring teacher on this generated critique.

Datasets

The study utilizes several challenging reasoning benchmarks:

  • Knights & Knaves (K&K): Logic puzzles involving 4–8 characters who either always lie or always tell the truth.
  • DeepMath: Competition-level mathematics problems categorized into difficulty bands (L5-7 and L7-10).
  • ProcessBench: A benchmark of human-annotated error steps used to validate that their findings hold true on human-labeled data, not just model-labeled data.

Key Insights and Ideas

  • Scoring is not Searching: Simply having the answer in context doesn't mean the model "sees" the mistake. Locating an error is a reasoning problem that requires an explicit search. As shown in Figure 1, the standard reference-conditioned teacher barely reacts to errors.
Figure 1: The reference-conditioned teacher barely reacts at the student’s errors. Content KL (top) and ∆log p of the student’s token (bottom) around gold error sentences (blue) and clean sentences (orange), aligned to sentence onset and end; Qwen3-1.7B scoring its own rollouts. Bands: 95% CI.
  • Generation Triggers the Search: The same model that fails to provide a corrective signal via scoring can often identify the error perfectly if asked to write a critique.
  • Backtracking Markers: When conditioned on a critique, the teacher doesn't just lower the probability of the wrong token; it redirects that probability toward "self-correction" tokens like "Wait", "But", or "Actually".
  • The Power of the Quote: Ablation studies show that the verbatim quote in the critique is what drives localization (pointing to the error), while the reasoning in the critique drives correction (explaining how to fix it).

Key Results

  • Failure of Standard OPSD: Standard reference-conditioned signals were found to be no better than chance at distinguishing correct rollouts from incorrect ones within the same problem (Figure 2).
Figure 2: The reference-conditioned signal cannot tell correct rollouts from wrong ones. Pooled AUC of trace-mean ∆log p, correct vs. wrong rollouts, by privileged context and teacher.
  • Success of Critique-Conditioning: By using a critique, the "Error-span Z" (a measure of signal strength at the error) becomes significantly positive for localization and negative for correction across all tasks (Figure 3/5).
Figure 3: With a critique in context, the same teacher reacts at the errors. Same layout as Figure 1, with gpt-5.2 critiques in place of the reference; Qwen3-1.7B scoring its own rollouts. The error curve leaves its control on every panel.
  • Test-Time Gains: On math problems the model couldn't solve by simple sampling, resuming a failed rollout with a critique in the context allowed the model to solve the problem twice as often as the control group (Table 5).
  • RL Training: In "cold-start" scenarios where standard GRPO (Group Relative Policy Optimization) provides no signal because all rollouts are wrong, the critique-conditioned signal provides a usable advantage, leading to higher out-of-distribution accuracy.

Limitations and Future Work

  • Compute Overhead: Generating a critique for every wrong rollout adds a computational cost (roughly 6% to 22% increase in step time depending on the critic model).
  • Scale: The training experiments were conducted on a relatively small scale (320 puzzles). The authors note that full-scale RL runs are still needed.
  • Privileged Information: The critiques in this study were generated using the reference solution. While this works for distillation, the authors emphasize that this doesn't prove the model can yet correct itself without that privileged information. Future work could explore "unaided" self-critique during RL.

Reference Table: Table 1: Quantitative comparison of Error-span Z across models and tasks.

Table 1: Error-span Z at the first gold error, for three students each scoring its own rollouts. Localization: log content KL (positive = reacts at the error); correction: ∆log p of the student’s token (negative = corrective). Bold: 95% CI excludes 0.

Summary updated

Abstract

On-policy self-distillation promises to replace the sparse outcome signal of verifiable-reward RL with dense token-level credit: a teacher conditioned on the reference solution scores the student's own rollout, and the resulting distributional shift is distilled back into the policy. We investigate whether this scoring signal actually contains corrective credit. Using gold error spans in student rollouts, we test the two signals consumed by self-distillation objectives, the full-distribution KL and the token log-ratio, for two properties: whether they localize errors and whether they push probability away from erroneous student tokens. They exhibit neither of these properties nor can they reliably distinguish correct from incorrect rollouts within the same problem. We argue that this failure is structural: locating an error in a reasoning trace is itself a search problem, while a scoring teacher is never explicitly asked to perform that search. The same model, with the same privileged information, localizes errors substantially better when allowed to generate a critique. This forms the motivation for \emph{critique-conditioned self-distillation}: first generate a critique of the current rollout, then condition the scoring teacher on that critique. The resulting signal localizes errors, is corrective within error spans, and shifts probability toward tokens that initiate backtracking such as ``Wait'', ``But'', and ``Actually''. We further show that the critique is useful both at test time, where resuming a wrong rollout with the critique in context converts math problems the model cannot solve by sampling, and in cold-start GRPO, where it provides an advantage signal on prompts for which verifiable rewards alone provide none.
BibTeX
@misc{maniparambil2026privileged,
  author = {Mayug Maniparambil and Ayush Rai and Ekansh Verma and Noel E. O'Connor and Fergal Reid},
  title = {{A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning}},
  year = {2026},
  eprint = {2610.13744},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.13744},
}
Figure