When a small language-model agent is trained to imitate a large one, it learns the large model's confidence along with its behavior. In multi-turn tasks such as navigating a household simulator or shopping on a website, that confidence becomes a liability once the student makes its first mistake. The student then lands in situations the teacher never visited. The teacher still issues sharp, near-certain action recommendations there, but nothing guarantees those recommendations are right. The student copies them, commits the resulting actions to its history, and later turns build on them. We propose to measure, for each state the student actually reaches, whether the teacher can still finish the task from there. We do this by letting the teacher take over and play the episode out. That outcome-based reliability score then decides how hard the student is pushed toward the teacher's choice at that turn.
The core bet fits in two sentences. In on-policy distillation, the teacher's own confidence is least trustworthy exactly in the off-track states where the student most needs guidance, so gating supervision on teacher entropy fails for the same reason distillation induces overconfidence. Replacing that self-reported confidence with an empirical "can the teacher still win from here?" signal should cut cascading failures more than either fixed schedules or entropy-based gating do.
Background and the gap
Two literatures meet here. The first studies distillation-induced overconfidence: a student is trained to match a teacher's probability outputs, and when those outputs are sharp and miscalibrated, the student inherits the sharpness without the competence. Revisit the Essence of Distilling Knowledge through Calibration reports a Spearman correlation of about −91% between teacher calibration error and student accuracy under standard probability-matching distillation. That correlation nearly disappears under ranking-based distillation. TRUST THE UNCERTAIN TEACHER reshapes the teacher's entropy on hard inputs and clips wrong-class probability mass, gaining up to +4.27 points over vanilla distillation on intent classification. Toward Understanding Adversarial Distillation shows that forcing a small student to match confident teacher predictions on inputs it cannot fit makes it memorize noise. All three work in closed-set classification, which leaves the problem open for sequential decisions. Quantifying Faithful Confidence Expression in Large Reasoning Models moves to generation. It finds that distilled reasoning models sound decisive without being accurate, and that prompting them to hedge has near-zero effect. It evaluates only isolated problems. Its prefix-conditioned estimator, which freezes a reasoning prefix and samples continuations, is the methodological seed for our takeover signal.
The second literature studies cascading error propagation: a model treats its own earlier outputs as true premises. Investigating and Mitigating the Multimodal Hallucination Snowballing shows that vision-language models drop at least 31% in accuracy when an erroneous prior turn is placed in their history. LLM-as-RNN frames committed context as immutable. Improving Dialogue State Tracking with Turn-based Loss Function upweights early turns because early errors zero out all later dialogue-tracking accuracy. AgentBoard documents open models stalling after roughly six steps. These papers motivate the problem; none of them involves distillation.
The closest work is On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents, which we call Guided-OPD. It diagnoses rollout drift when distilling Qwen3-30B-A3B into 0.6B to 4B students. Its fix has two parts. First, at each turn, either the teacher or the student acts, with the teacher's share decaying on a fixed cosine schedule. Second, the student is pushed toward the teacher's dominant action (reverse KL) on its own turns, and toward covering the teacher's whole distribution (forward KL) on teacher turns. It treats the teacher's outputs as correct everywhere and names uncertainty-aware schedules as future work. Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation (OCSD) restricts updates to high-NLL steps, but it is self-distillation inside RL, and its uncertainty signal is the model's own likelihood. Our proposal differs from both in one specific way. Neither measures whether the teacher's confidence at a student-visited state is earned, and we make that measurement the control signal.
Core Claim
Consider the student-visited states that lie far from the teacher's own trajectories. Our hypothesis is that, in these states, the teacher's action confidence decouples from its actual ability to complete the task. Because of this decoupling, mode-seeking distillation on these states transfers confident wrong actions and lowers the student's ability to recover after a first error. A calibrator trained on teacher takeover outcomes predicts teacher reliability better than teacher entropy does. Using it to modulate the distillation loss per turn should improve success rate and, more specifically, post-error recovery, at matched compute.
Research Program
Diagnostic stage. We run Guided-OPD students in ALFWorld, ScienceWorld and WebShop and sample states $s_t$ from their rollouts. From each sampled state, the teacher takes over and completes the episode $k$ times. The fraction of successful completions is the empirical teacher reliability $r(s_t)$. We then plot teacher confidence (top-action probability and entropy) against $r(s_t)$, binned by distance from the teacher's own state distribution. Distance is measured by turns since the student's first divergence from a teacher rollout. The first prediction is that the calibration gap between confidence and $r$ widens with this distance. We also define the student's recovery rate as the fraction of episodes that still succeed after the first action the teacher would have rejected. We relate it to the student's action-level overconfidence. Where agents emit free-text thoughts, we score the thoughts for decisiveness with an LLM judge in the manner of Gani et al. This tests whether assertive wording precedes unrecoverable cascades. That last analysis is secondary.
Method stage. Takeovers are expensive, so we run them on a small fraction of turns, around 10%. We train a lightweight probe on the teacher's hidden states, similar to the representation-based confidence estimator in Gani et al., to predict $r$; call its prediction $\hat r(s)$. The student's per-turn loss is
$$ \mathcal{L}_t = \hat r(s_t)\, D_{\mathrm{KL}}\big(\pi_S(\cdot\mid s_t)\,\|\,\pi_T(\cdot\mid s_t)\big) + \big(1-\hat r(s_t)\big)\, D_{\mathrm{KL}}\big(\pi_T^{(\tau)}(\cdot\mid s_t)\,\|\,\pi_S(\cdot\mid s_t)\big), $$
where $\pi_S$ and $\pi_T$ are the student and teacher action distributions and $\pi_T^{(\tau)}$ is the teacher softened at temperature $\tau>1$. When the teacher can still win from a state, the student is pushed onto the teacher's dominant action. When it cannot, the student is only asked to keep the teacher's broader set of options in play, rather than commit to a mode the evidence says is unreliable. The same $\hat r$ also replaces the cosine schedule. The teacher is handed control in proportion to its reliability, so it intervenes where it can actually rescue the episode.
Baselines
The naive reference is vanilla on-policy distillation, with reverse KL on student rollouts and no teacher mixing. It establishes how large the drift problem is in our setup.
The strongest existing method is Guided-OPD (On-Policy Distillation with Curriculum Turn-level Guidance). It reports average gains over vanilla on-policy distillation of +21.1% in Score and +25.5% in Success Rate across ALFWorld, ScienceWorld and WebShop, rising to +35.3% Score and +32.8% Success Rate for the 0.6B student. No reported result was found on Lacuna for Guided-OPD's absolute per-benchmark success rates. We therefore reimplement it with the same Qwen3-30B-A3B teacher and Qwen3-0.6B, 1.7B and 4B students, use identical environment splits, and tune its decay length and mixing ratio with the same number of tuning runs we spend on $\tau$ and probe size. Beating it means a higher success rate on all three environments for at least two student sizes, outside seed variance over three seeds. Recovery rate must also improve, since that is the mechanism the proposal claims to fix.
Fair compute is essential because takeovers cost teacher inference. Every baseline therefore receives the same total teacher-forward-token budget. For Guided-OPD and vanilla on-policy distillation, the extra budget goes to more training rollouts.
Three ablations isolate the key ingredient. In the first, $\hat r$ is replaced by a function of teacher entropy, which is the natural "uncertainty-aware" extension Guided-OPD names and the signal family OCSD uses. In the second, $\hat r$ is replaced by a global constant tuned on validation, which amounts to a fixed mix of reverse and forward KL with temperature scaling. The third keeps the reliability-weighted loss but retains the cosine schedule, which separates the loss change from the control-handoff change.
For context only, we report GRPO and OCSD numbers. OCSD reports 87.2% ALFWorld success at 8B against 73.2% for GRPO (Observation-Calibrated Self-Distillation). That is a different training regime and model size, so it is not a head-to-head target.
Why not just…?
Why not just temperature-scale the teacher before distillation? A single temperature softens every state equally, but the diagnostic predicts the miscalibration is state-dependent: confident and correct on-track, confident and wrong off-track. Fan et al. and Kim et al. both argue that uniform rescaling cannot fix structured miscalibration, even in classification. The constant-$\hat r$ ablation is exactly this alternative, tuned. If it matches the full method, the per-state signal is not pulling its weight, and we would report that.
Why not just gate supervision on teacher or student entropy? This is the real threat. It is cheap and needs no takeovers. Our argument against it rests on the claim that teacher entropy carries little information about reliability off-track, which is what distillation-induced overconfidence would predict. Gani et al. found that logit-peakiness confidence tracks decisive wording ($r=0.631$) while models stay decisive on tasks they often fail. That evidence is suggestive but comes from a different setting. The diagnostic stage settles the question directly by comparing the predictive power of entropy against the takeover probe for $r(s)$. If entropy predicts reliability nearly as well, we drop takeovers and the contribution shrinks to entropy-gated distillation.
Why not just upweight early turns, as the turn-based dialogue loss does? Early errors do dominate cascades. But uniformly upweighting turn 1 also amplifies the teacher's confident guesses on ambiguous opening states. These are exactly the inputs a small student may be unable to fit, the failure described in the adversarial distillation work. We include a turn-weighted on-policy distillation variant using the $w_t = 1 + (T-t)\lambda$ schedule from Improving Dialogue State Tracking with Turn-based Loss Function. We then check whether its gains concentrate on low- or high-reliability early states.
Why not just train with outcome-reward reinforcement learning and skip distillation? RL avoids imitating a miscalibrated teacher altogether. However, sparse rewards make credit assignment hard, which is the problem OCSD was built to address, and small students rarely succeed often enough to learn from. Takeover outcomes are in fact a teacher-assisted value estimate. A compute-matched GRPO run on the same student tells us whether distillation adds anything once that signal exists.
Proposed Experiments
The decisive checks follow in order. The first is whether the teacher's confidence-versus-takeover-success gap grows with distance from its own trajectories. If it does not, the premise fails and we stop. The second is whether the takeover probe beats entropy at predicting $r(s)$, measured by AUROC and calibration error on held-out states. The third is whether, at matched teacher compute, the full method beats tuned Guided-OPD on success rate, AgentBoard-style progress rate, and post-error recovery rate across ALFWorld, ScienceWorld and WebShop. We expect the gains to be largest for the 0.6B student, where Guided-OPD itself found drift most severe. The fourth is whether the improvement localizes to low-$\hat r$ states, which we test by stratifying recovery by the reliability of the state where the first error occurred.
A success on the third check without the fourth would mean the method helps for reasons other than the one claimed, and we would say so.
References
- Revisit the Essence of Distilling Knowledge through Calibration
- TRUST THE UNCERTAIN TEACHER: DISTILLING DARK KNOWLEDGE VIA CALIBRATED UNCERTAINTY
- Toward Understanding Adversarial Distillation: Why Robust Teachers Fail
- Quantifying Faithful Confidence Expression in Large Reasoning Models
- Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models
- LLM-as-RNN: A Recurrent Language Model for Memory Updates and Sequence Prediction
- Improving Dialogue State Tracking with Turn-based Loss Function and Sequential Data Augmentation
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
- On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents
- Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation