The Role of Feedback Alignment in Self-Distillation

TL;DR

This paper demonstrates that for self-distillation to be effective, feedback must be structurally aligned with a model's own reasoning. By using a "step-aligned" critique—where a critic copies a student's correct steps verbatim and only fixes the errors—researchers achieved a 16.11-point gain over standard reinforcement learning (GRPO), effectively providing "implicit process supervision" without the need for manual step-level labels.

What Problem Does This Paper Address?

Post-training is the "secret sauce" that turns base language models into capable reasoners. Currently, two main methods dominate:

  1. Reinforcement Learning from Verifiable Rewards (RLVR): Models learn from a simple "correct/incorrect" signal. While easy to implement, it suffers from poor credit assignment—the model knows it failed but doesn't know exactly where in a 500-token chain of thought the mistake happened.
  2. Distillation: A smaller student model learns to mimic a larger teacher. This provides dense, token-level signals but is often expensive and limited by the teacher's proprietary nature.

Self-distillation is a middle ground where a model acts as its own teacher when given extra context (like a hint or a reference solution). However, we don't actually know what kind of context makes for the best "self-teacher." This paper investigates how the structure and granularity of feedback affect what a model actually learns.

Methods

The authors set up a solver–critic framework using a small solver (Qwen3-1.7B) and a larger, frozen critic (QwQ-32B). They compared three training conditions:

  • GRPO (Baseline): Standard reinforcement learning using only a binary reward for the final answer.
  • REFSOL (Reference Solution): The self-teacher is shown the question plus a gold-standard reference solution.
  • STEPALIGNFB (Step-Aligned Feedback): The critic analyzes the student's specific reasoning trace. It follows a "faithful-scribe" convention: it copies the student's correct steps verbatim and only provides a correction starting at the first erroneous step.

The training objective for the self-distillation variants was to minimize the divergence (Forward KL) between the "student" (model seeing only the question) and the "self-teacher" (model seeing the question + feedback).

Datasets

The study utilized a difficulty-filtered subset of OpenMathReasoning.

  • Filtering Strategy: To ensure the experiment was meaningful, they selected problems that were "hard" for the 1.7B model (low initial accuracy) but solvable by the 32B critic.
  • Size: The final dataset consisted of 312 samples (282 for training, 30 for evaluation). While small, this allowed for deep per-token analysis of the learning signal.

Key Insights and Ideas

  • Alignment Matters More Than "Gold" Quality: Surprisingly, providing a perfect reference solution (REFSOL) is less effective than a tailored critique. Because a reference solution uses different wording and logic than the student, the distillation signal becomes "diffuse." The model gets penalized for stylistic differences even on steps where its logic was perfectly correct.
  • The "Faithful Scribe" and Induction Heads: The authors found that copying the student's correct steps verbatim is crucial. Mechanistically, this leverages the model's induction heads (circuits that excel at prefix-matching and copying). By anchoring the teacher's distribution to the student's correct prefix, the "advantage" signal is concentrated precisely at the point of failure.
  • Implicit Process Supervision: Usually, "process supervision" requires training a separate Reward Model (PRM) on thousands of human-labeled steps. This paper shows that step-aligned textual feedback provides a similar "dense" signal automatically, telling the model exactly which tokens to change.

Key Results

The results show a clear hierarchy in training effectiveness (measured in Avg@12 accuracy):

Method Avg@12 Accuracy Gain over GRPO
GRPO (RL Baseline) 19.72 -
REFSOL (Ref. Solution) 30.56 +10.84
STEPALIGNFB (Aligned) 35.83 +16.11

Qualitative Findings: * STEPALIGNFB outperformed REFSOL by 5.27 points, proving that how you show the answer is as important as the answer itself. * Per-token analysis (see Figure 3 and 4 in the paper) confirmed that aligned feedback creates sharp "negative advantages" at errors, whereas reference solutions create "noisy" signals across the entire response.

Limitations and Future Work

  • Scope: The study was limited to the mathematical domain and a single dataset (OpenMathReasoning). It remains to be seen if these "faithful-scribe" benefits translate to creative writing or coding.
  • Critic Cost: Generating high-quality, step-aligned critiques requires a significantly more powerful model than the solver. This increases the computational overhead of the training pipeline compared to simple RL.
  • Future Directions: The authors suggest looking for "cheaper" ways to generate these critiques or investigating if the model can eventually provide this aligned feedback to itself without a larger critic (true self-correction).

Summary updated

Abstract

Language models often perform better when their prompts include helpful context, such as feedback on a previous attempt. Self-distillation trains the model to achieve this performance without the context at inference time. It matches the model's output distribution across two settings: a student that sees only the question, and a self-teacher that also sees the context. What the model learns depends on the context that the self-teacher receives, yet its design (e.g. its structure, granularity, and alignment with the student's reasoning) remains largely unexplored. We study feedback design by training a solver on feedback from a frozen critic. We compare three conditions: (i) a binary reward (GRPO), (ii) the reference solution, and (iii) a step-by-step critique aligned to the solver's reasoning trace. Step-aligned critique yields the largest gains, outperforming GRPO by 16.11 points and outperforming reference-solution-conditioned self-distillation by 5.27 points (in Avg@12). Per-token advantage analysis explains why: step-aligned feedback concentrates distributional shifts at the tokens where reasoning fails, acting as implicit process supervision. In contrast, the reference solution spreads signal across all tokens, because an alternative derivation diverges from the solver's trace even at correct steps. Our results suggest that structural alignment between feedback and the solver's reasoning is one of the main drivers of self-distillation effectiveness.
BibTeX
@inproceedings{2026role,
  title = {{The Role of Feedback Alignment in Self-Distillation}},
  booktitle = {RLxF @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=EGxnTm42YG},
}
Figure