When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning

Xiaogeng Liu, Xinyan Wang, Yingzi Ma, Yechao Zhang, Chaowei XiaoarXiv 2026 · arXiv

TL;DR

Researchers have discovered that in mathematical reasoning, a "teacher" model's reliability isn't uniform; it is significantly less reliable at the beginning of a reasoning chain than at the end. By simply weighting distillation losses to favor later tokens—a method called Position-Weighted On-Policy Self-Distillation (PW-OPSD)—they achieved consistent performance gains in complex reasoning tasks across multiple model families without any extra computational overhead.

What Problem Does This Paper Address?

On-Policy Self-Distillation (OPSD) is a popular technique for training models to reason. In this setup, a "student" model generates a reasoning path, and a "privileged teacher" (often the same model, but provided with the ground-truth answer) provides token-level targets to guide the student.

The standard approach treats every teacher-provided token as equally valuable. However, reasoning is a sequential process where one wrong turn can derail the entire logic. Previous attempts to fix this used entropy (local uncertainty) to downweight tokens, but the authors argue this is flawed. High entropy in reasoning can mean two very different things:

  1. Non-viable uncertainty: The teacher is confused and about to make a mistake.
  2. Benign diversity: There are multiple valid ways to solve the problem (e.g., different wording or valid alternative steps).

Distinguishing between "I'm lost" and "There are many ways to do this" is the core challenge this paper solves.

Methods

The authors first developed a branch-viability diagnostic to probe teacher reliability. They took successful reasoning paths, identified points of high teacher ambiguity, and "forced" the model to take alternative paths. They then checked if the model could still reach the correct answer from those forced branches.

Figure 1: Branch viability reveals a positional reliability structure in Qwen3-4B reasoning traces. Alternatives are selected by the same model under the privileged-answer teacher prompt and rolled out under the ordinary student prompt; for high-ambiguity candidate positions, these forced student- context continuations fail more often at early positions and rarely fail late in the trace. (A) Sorting candidates by normalized position therefore separates teacher-unreliable branch points from viable diversity, whereas local uncertainty (entropy) scores yield near-flat failure curves. (B) The resulting empirical reliability curve motivates the PW-OPSD weight schedule, which discounts early high- ambiguity positions while preserving full-strength supervision later. See section 3.2 for the details of the diagnostic experiments.

Their key finding was that position is the best predictor of whether an ambiguous token is reliable. Early in a trace, ambiguity usually signals a derailment. Late in a trace, it usually signals benign diversity.

Based on this, they proposed PW-OPSD. Instead of a uniform weight, they apply a sigmoid-based weight schedule: $$w_{i,t} = w_{min} + (1 - w_{min})\sigma\left(\frac{r_{i,t} - \tau}{s}\right)$$ where $r_{i,t}$ is the normalized position in the sequence. This schedule discounts early tokens (where the teacher is "suspicious") and preserves full-strength supervision for later tokens.

Datasets

The models were trained on the OpenThoughts-Math-30k dataset. Evaluation was conducted across several challenging mathematical reasoning benchmarks:

  • MATH-500: A standard subset of the MATH dataset.
  • AIME 2024 & 2025: Problems from the American Invitational Mathematics Examination, known for high difficulty.
  • HMMT 2025: Problems from the Harvard-MIT Mathematics Tournament.

Key Insights and Ideas

  • Position > Entropy: Local uncertainty (entropy) is a poor predictor of teacher reliability (AUROC $\le$ 0.57). In contrast, the normalized position in the reasoning chain is a powerful predictor (AUROC 0.83).
  • The "Early Derailment" Phenomenon: Reasoning traces have a specific structural vulnerability. If a model is going to fail due to ambiguity, it almost always happens in the first half of the response. By the time it reaches the end, if it's still on track, local variations in tokens don't hurt correctness.
  • Zero-Cost Improvement: Unlike other adaptive distillation methods that require multiple teacher passes or auxiliary verifiers, PW-OPSD is a simple loss-weighting change that adds zero computational cost to the training process.
Table 1: Branch-viability classification on Qwen3-4B. Entries are within-problem residualized AUROC for separating 8 real-uncertain candidates (whose forced student-context continuations fail to recover the correct answer) from 271 diversity candidates (whose continuations remain viable) across 61 problems, restricted to the correct-spine subset and to high-truncated-entropy candidate positions. For position, the scored feature is 1 −er, so larger values correspond to earlier tokens. Cluster-bootstrap 95% CIs are computed over problems.

Key Results

PW-OPSD showed consistent improvements over standard OPSD and other entropy-based baselines (like EOPD and REOPOLD):

  • AIME Improvements: On Qwen3-4B, PW-OPSD improved the Avg@12 score by +1.0 points on AIME 2024 and +1.1 points on AIME 2025.
  • HMMT Gains: The "Aggressive" weight schedule improved HMMT 2025 performance by +1.48 points.
  • Cross-Model Generalization: The benefits weren't limited to Qwen. The same position-weighting schedule improved Avg@12 for DeepSeek-R1-Distill-Llama-8B (+0.35 points) and Olmo-3-7B-Think (+0.50 points).
Table 2: Qwen3-4B results with maximum generation length 38,912 tokens. Entries report mean ± standard deviation across three evaluation seeds. Bold marks the best value for each benchmark– metric column and any value within 0.1 pp of the leader. The Aggressive row is included to show schedule sensitivity rather than as a separate method claim.

Limitations and Future Work

The authors acknowledge that while PW-OPSD is effective and easy to implement, its gains are relatively modest. They suggest that future work could move beyond simple position-based heuristics toward more sophisticated, position-conditioned objectives—such as dynamically mixing forward and reverse KL divergence based on the model's progress through a reasoning trajectory. Additionally, the current method relies on a fixed sigmoid schedule; learning an optimal weighting function per task could yield further improvements.

Summary updated

Abstract

On-policy self-distillation (OPSD) trains a student on its own rollouts using a privileged teacher, but its standard objective weights all generated tokens equally, implicitly treating the privileged teacher target as equally reliable at every student-visited prefix. Existing entropy-based OPD methods relax this uniformity by modulating token-level supervision with teacher entropy, but high teacher entropy in reasoning has an ambiguous reliability meaning: it can reflect either non-viable uncertainty or benign solution diversity. To identify this phenomenon, we introduce a branch-viability diagnostic. Specifically, we record next-token alternatives from the privileged-answer teacher prompt, force each alternative after the student prompt plus its on-policy spine prefix, and test whether the resulting student-template continuation recovers the correct answer. On Qwen3-4B, we find that an oriented within-sequence position score is the strongest tested predictor of teacher-token reliability, reaching an area-under-ROC-curve (AUROC) of 0.83; local uncertainty scores are at most 0.57. Motivated by this trajectory-level structure, we propose Position-Weighted On-Policy Self-Distillation (PW-OPSD), which applies an increasing position weight while keeping the same student rollout, privileged teacher pass, and clipped forward-KL target as OPSD. In our comprehensive evaluations with different random seeds, the diagnostic-derived PW-OPSD improves AIME 2024 and AIME 2025 Avg@12 by +1.0 and +1.1 points, and a generalization evaluation on two larger-scale models from different families, DeepSeek-R1-Distill-Llama-8B and Olmo-3-7B-Think, also demonstrates consistent aggregate Avg@12 improvements. These results show that teacher-token reliability in reasoning distillation is trajectory-structured and can be utilized without additional teacher computation.
BibTeX
@misc{liu2026when,
  author = {Xiaogeng Liu and Xinyan Wang and Yingzi Ma and Yechao Zhang and Chaowei Xiao},
  title = {{When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning}},
  year = {2026},
  eprint = {2605.21606},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.21606},
}
Figure