When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation

Yinuo Jiang, Yongjie Ye, Zhou Tao, Xiang Zhuang, Qiang Zhang, Huajun Chen, Tiankai LiarXiv 2026 · arXiv

TL;DR

On-Policy Distillation (OPD) helps smaller models learn from larger ones, but teacher models often pass down input-agnostic biases—like formatting preferences or language priors—instead of task-relevant knowledge. This paper introduces SA-OPD, a framework that identifies and filters out these "spurious signals" by checking if the teacher's feedback changes when the input prompt is removed. By focusing exclusively on input-grounded supervision, SA-OPD significantly boosts performance across both language and vision-language tasks with minimal computational overhead.

What Problem Does This Paper Address?

Knowledge distillation transfers capabilities from a large, powerful "teacher" model to a smaller "student" model. On-Policy Distillation (OPD) optimizes this process dynamically: it lets the student generate its own trajectories and applies token-by-token teacher supervision on those student-visited states.

However, teacher models are not infallible truth-tellers. Their token-level judgments are often shaped by input-agnostic language priors, formatting conventions, or stereotyped reasoning templates acquired during pre-training. For example, a teacher might heavily reward a generic phrase or penalize a valid student response simply because it deviates from the teacher's preferred stylistic template, regardless of the actual input task.

Existing token-selection methods prioritize teacher signals that are confident, divergent, or learnable, but they completely overlook whether the signal is actually grounded in the input. These ungrounded, high-impact spurious signals introduce low signal-to-noise ratio (SNR) gradients, causing the student to inherit the teacher's superficial biases rather than actual task-solving capabilities.

Figure 1: Illustration of spurious signals in OPD. Panels (a) and (b) present a concrete example: (a) shows the input image and question, while (b) visualizes the token-level teacher–student divergence along the corresponding student rollout. Some tokens exhibit nearly unchanged teacher–student divergence after prompt removal, suggesting that their dense supervision is driven by generic entity collocations or prior templates rather than input-specific visual evidence. Panel (c) provides the gradient interpretation: such weakly grounded signals induce low-SNR OPD updates, with large prior-driven gradient components but weak alignment with the ideal task-improving direction.

Methods

The authors propose SA-OPD (Spurious-Signal-Aware On-Policy Distillation) to isolate and filter out these misleading updates.

The core mechanism relies on estimating an Input-Grounding Gap ($\Delta I G_t$). For every token generated during an on-policy rollout, SA-OPD computes the teacher-student divergence under two distinct conditions:

  1. Full-Prompt Context ($A^{\text{full}}_t$): The standard divergence calculated with the original input prompt.
  2. Residual No-Prompt Context ($A^{\text{res}}_t$): The divergence calculated after completely removing the input prompt while keeping the student's generated prefix.

If the teacher's feedback barely changes when the prompt is stripped away (resulting in a low $\Delta I G_t$), the signal is deemed a spurious prior.

SA-OPD then applies a Token-level Filter that targets the intersection of two conditions: low input-groundedness and high optimization impact (large absolute divergence). To prevent the filter from being overly aggressive and discarding useful supervision in unstable tasks, the authors introduce an adaptive Dynamic FLMR (Filtered Loss-Mass Ratio) Constraint to bound the total proportion of loss mass removed per batch.

Figure 2: Overview of the proposed SA-OPD framework. Given an on-policy rollout, SA-OPD computes teacher–student divergence under the original prompt and a residual no-prompt context. If the divergence barely changes after prompt removal, the teacher signal is likely dominated by input-agnostic priors. SA-OPD filters tokens that are both weakly input-grounded and high-impact, enabling OPD to update from input-grounded distillation signals rather than spurious signals.

Datasets

The framework was evaluated across both Large Language Model (LLM) and Vision-Language Model (VLM) distillation setups:

  • LLM Distillation: Trained using a 7K math-reasoning subset from DeepMath. Evaluation spanned five benchmarks: Math500, AMC 2023, AIME 2024, AIME 2025, and MinervaMATH.
  • VLM Distillation: Trained on sampled subsets from VERO-600K (for visual understanding) and MMRL30k (for visual reasoning). Evaluation covered six benchmarks: EvoChart, MMIFEval, CountQA (visual understanding) and MathVision, Geo3K, MathVista (visual reasoning).

Key Insights and Ideas

  • Teachers have bad habits: A significant portion of high-impact teacher feedback is driven by local textual context and template preferences rather than actual task evidence.
  • The "No-Prompt" baseline is highly effective: Simply removing the prompt is a lightweight yet powerful proxy for isolating input-agnostic language priors from true task-driven feedback.
  • VLM vs. LLM dynamics differ structurally: In LLM mathematical reasoning, spurious signals drop to near zero after the initial alignment phase. In contrast, VLM training exhibits persistent spurious signals throughout the entire process because vision-language teachers must continuously juggle visual grounding, perceptual uncertainty, and language priors.
  • Spurious signals bypass entropy checks: Traditional filtering methods rely heavily on token entropy (uncertainty). However, SA-OPD reveals that a substantial fraction of filtered spurious tokens actually have low student entropy, meaning standard diagnostics completely miss them.

Key Results

  • Vision-Language Performance: SA-OPD achieved the best results across all six VLM benchmarks. Compared to Vanilla OPD, it improved the average score from 50.5 to 54.0 on visual understanding and from 60.4 to 63.5 on visual reasoning, outperforming strong selective baselines like TIP and FiRe-OPD.
  • Mathematical Reasoning Performance: On LLM math tasks, SA-OPD raised the average score from 28.5 to 30.4. Notably, it outperformed the strongest prior baseline on Math500 by 3.0 points.
  • Minimal Computational Overhead: Because the residual no-prompt scoring is only used to construct a filtering mask and requires no extra backward passes or trainable parameters, SA-OPD adds a modest training time overhead of only 2.64% to 7.53% across tasks.

Limitations and Future Work

  • Extra Forward Pass: Although computationally lightweight, the method still requires one additional forward pass for both the student and teacher models to evaluate the residual context.
  • Prompt-Dependent Templates: The residual context method assumes that removing the prompt entirely isolates the prior. However, if a specific prompt structure itself heavily dictates a formatting style, a completely blank prompt might not capture all template-driven biases.
  • Future Directions: The authors suggest extending input-groundedness evaluation to other alignment paradigms, such as Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO), where reward models might similarly suffer from input-agnostic biases.

Summary updated

Abstract

On-Policy distillation (OPD) transfers teacher capabilities by supervising student-sampled trajectories with dense token-level teacher signals. Recent selective OPD methods improve this process by prioritizing signals that are confident, informative, or learnable. However, the assumptions overlook a fundamental failure mode of language models: their token-level judgments can be driven by input-agnostic language priors, formatting conventions, or stereotyped reasoning templates rather than task-specific evidence. We refer to such optimization-relevant but weakly input-grounded supervision as spurious signals in OPD, which may produce large gradients while contributing little task-improving direction. To mitigate this issue, we propose SA-OPD, a Spurious-Signal-Aware On-Policy Distillation framework that identifies and filters misleading token-level supervision based on input-groundedness and optimization impact. SA-OPD introduces a lightweight input-groundedness proxy estimating whether a token-level distillation signal truly depends on the input. It then filters only tokens that simultaneously exhibit low input-groundedness and extreme distillation divergence, thereby removing high-impact spurious updates and achieving fine-grained OPD optimization. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that SA-OPD consistently outperforms Vanilla OPD and competitive selective methods. These results establish input-groundedness as a key dimension for OPD supervision selection and offer a simple, effective strategy for mitigating spurious updates.
BibTeX
@misc{jiang2026when,
  author = {Yinuo Jiang and Yongjie Ye and Zhou Tao and Xiang Zhuang and Qiang Zhang and Huajun Chen and Tiankai Li},
  title = {{When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation}},
  year = {2026},
  eprint = {2608.03632},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.03632},
}
Figure