TL;DR
Researchers have discovered a way to give Large Language Models (LLMs) the benefits of "process supervision"—the gold standard for teaching math reasoning—without the massive cost of training separate reward models or hiring humans to label every single step. By using a "step-aligned critique" during self-distillation, they created an implicit process reward signal that outperforms standard reinforcement learning (GRPO) by over 16 points and traditional reference-based distillation by 5 points.
What Problem Does This Paper Address?
Teaching an LLM to solve complex math is hard because of the "credit assignment" problem. If a model writes a 10-step proof and gets the wrong answer, where did it go wrong?
Standard reinforcement learning (like GRPO) uses outcome supervision: it just tells the model "the final answer is wrong." This is a coarse signal. The better way is process supervision, where every step is graded. However, building a Process Reward Model (PRM) is expensive; it requires thousands of human-labeled steps or complex "rollout" estimations.
The authors ask: Can we get the precision of a PRM using only natural language feedback in a simpler "self-distillation" pipeline?
Methods
The authors use a solver–critic setup.
- The Solver: A small model (Qwen3-1.7B) generates a reasoning trace.
- The Critic: A larger model (QwQ-32B) looks at the solver's trace and the ground-truth solution.
- Self-Distillation: The solver is trained to minimize the gap between its original prediction and a "teacher" version of itself that has access to the critic's feedback.
The "secret sauce" is STEPALIGNFB (Step-Aligned Feedback). Instead of just giving the solver a correct reference solution, the critic acts as a "faithful scribe." It repeats the solver's correct steps verbatim and only intervenes to fix the specific step where the solver tripped up. This keeps the feedback "aligned" to the solver's own logic.
Datasets
The study used a filtered subset of OpenMathReasoning.
- They specifically targeted "hard but tractable" problems—questions the 1.7B model usually fails but the 32B critic can solve.
- The final training set consisted of 282 problems, with 30 reserved for evaluation. While small, this allowed for a highly controlled analysis of feedback structure.
Key Insights and Ideas
- Alignment > Quality: Surprisingly, a perfect reference solution (REFSOL) is a mediocre teacher. Because the reference solution uses different notation or a different logical path than the solver, the distillation signal becomes "diffuse." The model gets penalized even for correct steps just because they don't look like the reference.
- The "Faithful Scribe" Convention: For distillation to work, the critic must repeat the student's correct steps exactly. This "anchors" the model.
- The Induction Head Connection: The authors propose a mechanistic reason for why this works. Transformers have "induction heads" that are very good at copying patterns from the context. By repeating the correct prefix, the critic "activates" these heads, concentrating the training signal (the "advantage") precisely on the tokens where the reasoning actually failed.
- Implicit PRM: This method effectively turns natural language into a dense, position-specific reward signal without ever training a scalar reward model.
Key Results
The researchers compared three signals: GRPO (binary reward), REFSOL (reference solution), and STEPALIGNFB (aligned critique).
- Accuracy (Avg@12): STEPALIGNFB reached 35.83%, beating GRPO (19.72%) and REFSOL (30.56%).
- Majority Vote: STEPALIGNFB showed a massive gain in "consensus" accuracy, suggesting it helps the model converge on correct reasoning paths rather than just getting lucky.
- Localization: Per-token analysis showed that STEPALIGNFB produced sharp "negative advantage" spikes exactly at the error points, whereas other methods spread the "blame" across the entire response.
Limitations and Future Work
- Critic Dependency: This method requires a "stronger" critic model to generate the feedback, which increases the computational cost during the training/data-generation phase.
- Scale: The study was conducted on a relatively small, filtered dataset. It remains to be seen how well this "faithful scribe" prompting scales to massive, diverse datasets or non-mathematical reasoning tasks.
- Cost vs. Performance: While it avoids training a PRM, generating high-quality, step-aligned critiques for every training rollout is still more expensive than simple binary verification. Future work may look for ways to generate these critiques more cheaply.
Summary updated