The Role of Feedback Alignment in Self-Distillation

Semih Kara, Oguzhan Ersoy

TL;DR

Researchers have discovered a way to give Large Language Models (LLMs) the benefits of "process supervision"—the gold standard for teaching math reasoning—without the massive cost of training separate reward models or hiring humans to label every single step. By using a "step-aligned critique" during self-distillation, they created an implicit process reward signal that outperforms standard reinforcement learning (GRPO) by over 16 points and traditional reference-based distillation by 5 points.

What Problem Does This Paper Address?

Teaching an LLM to solve complex math is hard because of the "credit assignment" problem. If a model writes a 10-step proof and gets the wrong answer, where did it go wrong?

Standard reinforcement learning (like GRPO) uses outcome supervision: it just tells the model "the final answer is wrong." This is a coarse signal. The better way is process supervision, where every step is graded. However, building a Process Reward Model (PRM) is expensive; it requires thousands of human-labeled steps or complex "rollout" estimations.

The authors ask: Can we get the precision of a PRM using only natural language feedback in a simpler "self-distillation" pipeline?

Methods

The authors use a solver–critic setup.

  1. The Solver: A small model (Qwen3-1.7B) generates a reasoning trace.
  2. The Critic: A larger model (QwQ-32B) looks at the solver's trace and the ground-truth solution.
  3. Self-Distillation: The solver is trained to minimize the gap between its original prediction and a "teacher" version of itself that has access to the critic's feedback.

The "secret sauce" is STEPALIGNFB (Step-Aligned Feedback). Instead of just giving the solver a correct reference solution, the critic acts as a "faithful scribe." It repeats the solver's correct steps verbatim and only intervenes to fix the specific step where the solver tripped up. This keeps the feedback "aligned" to the solver's own logic.

Datasets

The study used a filtered subset of OpenMathReasoning.

  • They specifically targeted "hard but tractable" problems—questions the 1.7B model usually fails but the 32B critic can solve.
  • The final training set consisted of 282 problems, with 30 reserved for evaluation. While small, this allowed for a highly controlled analysis of feedback structure.

Key Insights and Ideas

  • Alignment > Quality: Surprisingly, a perfect reference solution (REFSOL) is a mediocre teacher. Because the reference solution uses different notation or a different logical path than the solver, the distillation signal becomes "diffuse." The model gets penalized even for correct steps just because they don't look like the reference.
  • The "Faithful Scribe" Convention: For distillation to work, the critic must repeat the student's correct steps exactly. This "anchors" the model.
  • The Induction Head Connection: The authors propose a mechanistic reason for why this works. Transformers have "induction heads" that are very good at copying patterns from the context. By repeating the correct prefix, the critic "activates" these heads, concentrating the training signal (the "advantage") precisely on the tokens where the reasoning actually failed.
  • Implicit PRM: This method effectively turns natural language into a dense, position-specific reward signal without ever training a scalar reward model.

Key Results

The researchers compared three signals: GRPO (binary reward), REFSOL (reference solution), and STEPALIGNFB (aligned critique).

  • Accuracy (Avg@12): STEPALIGNFB reached 35.83%, beating GRPO (19.72%) and REFSOL (30.56%).
  • Majority Vote: STEPALIGNFB showed a massive gain in "consensus" accuracy, suggesting it helps the model converge on correct reasoning paths rather than just getting lucky.
  • Localization: Per-token analysis showed that STEPALIGNFB produced sharp "negative advantage" spikes exactly at the error points, whereas other methods spread the "blame" across the entire response.

Limitations and Future Work

  • Critic Dependency: This method requires a "stronger" critic model to generate the feedback, which increases the computational cost during the training/data-generation phase.
  • Scale: The study was conducted on a relatively small, filtered dataset. It remains to be seen how well this "faithful scribe" prompting scales to massive, diverse datasets or non-mathematical reasoning tasks.
  • Cost vs. Performance: While it avoids training a PRM, generating high-quality, step-aligned critiques for every training rollout is still more expensive than simple binary verification. Future work may look for ways to generate these critiques more cheaply.

Summary updated

Abstract

Process supervision (assigning credit to individual steps of a reasoning trace) has been shown to outperform outcome-only rewards for mathematical reasoning. The standard implementation trains a process reward model (PRM) on step-labeled data, which is expensive to collect. We show that using step-aligned critique as context in self-distillation provides PRM-like credit assignment. It correctly signs and localizes per-token advantages at error-prone steps in the solver's trace, without training a reward model. We compare three training signals in a solver--critic setup on a filtered subset of OpenMathReasoning: (i) a binary reward (\grpo), (ii) self-distillation with the reference solution, and (iii) self-distillation with a step-by-step critique aligned to the solver's reasoning trace. Step-aligned critique yields the largest gains, beating \grpo by 16.11 points in Avg@12 on the evaluation set and reference-solution-conditioned self-distillation by 5.27 points. Per-token advantage analysis explains why: step-aligned feedback concentrates distributional shifts at the tokens where reasoning fails, acting as implicit process supervision. In contrast, the reference solution spreads signal across all tokens, because an alternative derivation diverges from the solver's trace even at correct steps. Self-distillation with step-aligned critique therefore recovers the localization benefit of process supervision within a simpler training pipeline.
BibTeX
@inproceedings{kara2026role,
  author = {Semih Kara and Oguzhan Ersoy},
  title = {{The Role of Feedback Alignment in Self-Distillation}},
  booktitle = {RLxF @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=EGxnTm42YG},
}
Figure