TL;DR
Researchers have discovered that in mathematical reasoning, a "teacher" model's reliability isn't uniform; it is significantly less reliable at the beginning of a reasoning chain than at the end. By simply weighting distillation losses to favor later tokens—a method called Position-Weighted On-Policy Self-Distillation (PW-OPSD)—they achieved consistent performance gains in complex reasoning tasks across multiple model families without any extra computational overhead.
What Problem Does This Paper Address?
On-Policy Self-Distillation (OPSD) is a popular technique for training models to reason. In this setup, a "student" model generates a reasoning path, and a "privileged teacher" (often the same model, but provided with the ground-truth answer) provides token-level targets to guide the student.
The standard approach treats every teacher-provided token as equally valuable. However, reasoning is a sequential process where one wrong turn can derail the entire logic. Previous attempts to fix this used entropy (local uncertainty) to downweight tokens, but the authors argue this is flawed. High entropy in reasoning can mean two very different things:
- Non-viable uncertainty: The teacher is confused and about to make a mistake.
- Benign diversity: There are multiple valid ways to solve the problem (e.g., different wording or valid alternative steps).
Distinguishing between "I'm lost" and "There are many ways to do this" is the core challenge this paper solves.
Methods
The authors first developed a branch-viability diagnostic to probe teacher reliability. They took successful reasoning paths, identified points of high teacher ambiguity, and "forced" the model to take alternative paths. They then checked if the model could still reach the correct answer from those forced branches.
Their key finding was that position is the best predictor of whether an ambiguous token is reliable. Early in a trace, ambiguity usually signals a derailment. Late in a trace, it usually signals benign diversity.
Based on this, they proposed PW-OPSD. Instead of a uniform weight, they apply a sigmoid-based weight schedule: $$w_{i,t} = w_{min} + (1 - w_{min})\sigma\left(\frac{r_{i,t} - \tau}{s}\right)$$ where $r_{i,t}$ is the normalized position in the sequence. This schedule discounts early tokens (where the teacher is "suspicious") and preserves full-strength supervision for later tokens.
Datasets
The models were trained on the OpenThoughts-Math-30k dataset. Evaluation was conducted across several challenging mathematical reasoning benchmarks:
- MATH-500: A standard subset of the MATH dataset.
- AIME 2024 & 2025: Problems from the American Invitational Mathematics Examination, known for high difficulty.
- HMMT 2025: Problems from the Harvard-MIT Mathematics Tournament.
Key Insights and Ideas
- Position > Entropy: Local uncertainty (entropy) is a poor predictor of teacher reliability (AUROC $\le$ 0.57). In contrast, the normalized position in the reasoning chain is a powerful predictor (AUROC 0.83).
- The "Early Derailment" Phenomenon: Reasoning traces have a specific structural vulnerability. If a model is going to fail due to ambiguity, it almost always happens in the first half of the response. By the time it reaches the end, if it's still on track, local variations in tokens don't hurt correctness.
- Zero-Cost Improvement: Unlike other adaptive distillation methods that require multiple teacher passes or auxiliary verifiers, PW-OPSD is a simple loss-weighting change that adds zero computational cost to the training process.
Key Results
PW-OPSD showed consistent improvements over standard OPSD and other entropy-based baselines (like EOPD and REOPOLD):
- AIME Improvements: On Qwen3-4B, PW-OPSD improved the
Avg@12score by +1.0 points on AIME 2024 and +1.1 points on AIME 2025. - HMMT Gains: The "Aggressive" weight schedule improved HMMT 2025 performance by +1.48 points.
- Cross-Model Generalization: The benefits weren't limited to Qwen. The same position-weighting schedule improved
Avg@12for DeepSeek-R1-Distill-Llama-8B (+0.35 points) and Olmo-3-7B-Think (+0.50 points).
Limitations and Future Work
The authors acknowledge that while PW-OPSD is effective and easy to implement, its gains are relatively modest. They suggest that future work could move beyond simple position-based heuristics toward more sophisticated, position-conditioned objectives—such as dynamically mixing forward and reverse KL divergence based on the model's progress through a reasoning trajectory. Additionally, the current method relies on a fixed sigmoid schedule; learning an optimal weighting function per task could yield further improvements.
Summary updated