TL;DR
Standard self-distillation for LLM reasoning fails because simply scoring a trace with the answer in context doesn't actually pinpoint errors. This paper introduces Critique-Conditioned Self-Distillation, showing that forcing a model to first generate a critique creates a dense, corrective signal that successfully localizes mistakes and improves both test-time performance and reinforcement learning.
What Problem Does This Paper Address?
Reinforcement Learning with Verifiable Rewards (RLVR) is the gold standard for training reasoning models, but it suffers from "sparse" feedback—you only know if the final answer is right or wrong, not which specific step led to the failure. This is particularly problematic in "all-wrong groups," where every sampled attempt for a problem fails, leaving the model with no gradient to learn from.
To solve this, researchers use On-Policy Self-Distillation (OPSD). In this setup, a "teacher" (the model itself, but given the reference solution) scores the "student's" (the model without the answer) reasoning trace. The hope is that the teacher will provide "dense" feedback at every token. However, this paper discovers a structural flaw: a teacher that only scores a trace doesn't actually "search" for the error. Consequently, the feedback signal is often flat and fails to distinguish between correct and incorrect reasoning steps.
Methods
The authors investigate why the standard scoring signal is "empty" and propose a generative fix.
- Signal Analysis: They measure two primary signals:
- Localization (KL Divergence): Does the teacher's distribution change significantly at the exact moment an error occurs?
- Correction (Token Log-Ratio): Does the teacher actively push probability away from the student's erroneous tokens?
- Gold Error Spans: They use high-capability models (DeepSeek, Claude) to annotate exactly where reasoning traces go wrong, creating a "ground truth" for error localization.
- Critique-Conditioned Self-Distillation: Instead of just giving the teacher the reference solution, they ask the model to first generate a critique (a verbatim quote of the error plus an explanation). They then condition the scoring teacher on this generated critique.
Datasets
The study utilizes several challenging reasoning benchmarks:
- Knights & Knaves (K&K): Logic puzzles involving 4–8 characters who either always lie or always tell the truth.
- DeepMath: Competition-level mathematics problems categorized into difficulty bands (L5-7 and L7-10).
- ProcessBench: A benchmark of human-annotated error steps used to validate that their findings hold true on human-labeled data, not just model-labeled data.
Key Insights and Ideas
- Scoring is not Searching: Simply having the answer in context doesn't mean the model "sees" the mistake. Locating an error is a reasoning problem that requires an explicit search. As shown in Figure 1, the standard reference-conditioned teacher barely reacts to errors.
- Generation Triggers the Search: The same model that fails to provide a corrective signal via scoring can often identify the error perfectly if asked to write a critique.
- Backtracking Markers: When conditioned on a critique, the teacher doesn't just lower the probability of the wrong token; it redirects that probability toward "self-correction" tokens like "Wait", "But", or "Actually".
- The Power of the Quote: Ablation studies show that the verbatim quote in the critique is what drives localization (pointing to the error), while the reasoning in the critique drives correction (explaining how to fix it).
Key Results
- Failure of Standard OPSD: Standard reference-conditioned signals were found to be no better than chance at distinguishing correct rollouts from incorrect ones within the same problem (Figure 2).
- Success of Critique-Conditioning: By using a critique, the "Error-span Z" (a measure of signal strength at the error) becomes significantly positive for localization and negative for correction across all tasks (Figure 3/5).
- Test-Time Gains: On math problems the model couldn't solve by simple sampling, resuming a failed rollout with a critique in the context allowed the model to solve the problem twice as often as the control group (Table 5).
- RL Training: In "cold-start" scenarios where standard GRPO (Group Relative Policy Optimization) provides no signal because all rollouts are wrong, the critique-conditioned signal provides a usable advantage, leading to higher out-of-distribution accuracy.
Limitations and Future Work
- Compute Overhead: Generating a critique for every wrong rollout adds a computational cost (roughly 6% to 22% increase in step time depending on the critic model).
- Scale: The training experiments were conducted on a relatively small scale (320 puzzles). The authors note that full-scale RL runs are still needed.
- Privileged Information: The critiques in this study were generated using the reference solution. While this works for distillation, the authors emphasize that this doesn't prove the model can yet correct itself without that privileged information. Future work could explore "unaided" self-critique during RL.
Reference Table: Table 1: Quantitative comparison of Error-span Z across models and tasks.
Summary updated