TL;DR
On-Policy Distillation (OPD) helps smaller models learn from larger ones, but teacher models often pass down input-agnostic biases—like formatting preferences or language priors—instead of task-relevant knowledge. This paper introduces SA-OPD, a framework that identifies and filters out these "spurious signals" by checking if the teacher's feedback changes when the input prompt is removed. By focusing exclusively on input-grounded supervision, SA-OPD significantly boosts performance across both language and vision-language tasks with minimal computational overhead.
What Problem Does This Paper Address?
Knowledge distillation transfers capabilities from a large, powerful "teacher" model to a smaller "student" model. On-Policy Distillation (OPD) optimizes this process dynamically: it lets the student generate its own trajectories and applies token-by-token teacher supervision on those student-visited states.
However, teacher models are not infallible truth-tellers. Their token-level judgments are often shaped by input-agnostic language priors, formatting conventions, or stereotyped reasoning templates acquired during pre-training. For example, a teacher might heavily reward a generic phrase or penalize a valid student response simply because it deviates from the teacher's preferred stylistic template, regardless of the actual input task.
Existing token-selection methods prioritize teacher signals that are confident, divergent, or learnable, but they completely overlook whether the signal is actually grounded in the input. These ungrounded, high-impact spurious signals introduce low signal-to-noise ratio (SNR) gradients, causing the student to inherit the teacher's superficial biases rather than actual task-solving capabilities.
Methods
The authors propose SA-OPD (Spurious-Signal-Aware On-Policy Distillation) to isolate and filter out these misleading updates.
The core mechanism relies on estimating an Input-Grounding Gap ($\Delta I G_t$). For every token generated during an on-policy rollout, SA-OPD computes the teacher-student divergence under two distinct conditions:
- Full-Prompt Context ($A^{\text{full}}_t$): The standard divergence calculated with the original input prompt.
- Residual No-Prompt Context ($A^{\text{res}}_t$): The divergence calculated after completely removing the input prompt while keeping the student's generated prefix.
If the teacher's feedback barely changes when the prompt is stripped away (resulting in a low $\Delta I G_t$), the signal is deemed a spurious prior.
SA-OPD then applies a Token-level Filter that targets the intersection of two conditions: low input-groundedness and high optimization impact (large absolute divergence). To prevent the filter from being overly aggressive and discarding useful supervision in unstable tasks, the authors introduce an adaptive Dynamic FLMR (Filtered Loss-Mass Ratio) Constraint to bound the total proportion of loss mass removed per batch.
Datasets
The framework was evaluated across both Large Language Model (LLM) and Vision-Language Model (VLM) distillation setups:
- LLM Distillation: Trained using a 7K math-reasoning subset from DeepMath. Evaluation spanned five benchmarks: Math500, AMC 2023, AIME 2024, AIME 2025, and MinervaMATH.
- VLM Distillation: Trained on sampled subsets from VERO-600K (for visual understanding) and MMRL30k (for visual reasoning). Evaluation covered six benchmarks: EvoChart, MMIFEval, CountQA (visual understanding) and MathVision, Geo3K, MathVista (visual reasoning).
Key Insights and Ideas
- Teachers have bad habits: A significant portion of high-impact teacher feedback is driven by local textual context and template preferences rather than actual task evidence.
- The "No-Prompt" baseline is highly effective: Simply removing the prompt is a lightweight yet powerful proxy for isolating input-agnostic language priors from true task-driven feedback.
- VLM vs. LLM dynamics differ structurally: In LLM mathematical reasoning, spurious signals drop to near zero after the initial alignment phase. In contrast, VLM training exhibits persistent spurious signals throughout the entire process because vision-language teachers must continuously juggle visual grounding, perceptual uncertainty, and language priors.
- Spurious signals bypass entropy checks: Traditional filtering methods rely heavily on token entropy (uncertainty). However, SA-OPD reveals that a substantial fraction of filtered spurious tokens actually have low student entropy, meaning standard diagnostics completely miss them.
Key Results
- Vision-Language Performance: SA-OPD achieved the best results across all six VLM benchmarks. Compared to Vanilla OPD, it improved the average score from 50.5 to 54.0 on visual understanding and from 60.4 to 63.5 on visual reasoning, outperforming strong selective baselines like TIP and FiRe-OPD.
- Mathematical Reasoning Performance: On LLM math tasks, SA-OPD raised the average score from 28.5 to 30.4. Notably, it outperformed the strongest prior baseline on Math500 by 3.0 points.
- Minimal Computational Overhead: Because the residual no-prompt scoring is only used to construct a filtering mask and requires no extra backward passes or trainable parameters, SA-OPD adds a modest training time overhead of only 2.64% to 7.53% across tasks.
Limitations and Future Work
- Extra Forward Pass: Although computationally lightweight, the method still requires one additional forward pass for both the student and teacher models to evaluate the residual context.
- Prompt-Dependent Templates: The residual context method assumes that removing the prompt entirely isolates the prior. However, if a specific prompt structure itself heavily dictates a formatting style, a completely blank prompt might not capture all template-driven biases.
- Future Directions: The authors suggest extending input-groundedness evaluation to other alignment paradigms, such as Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO), where reward models might similarly suffer from input-agnostic biases.
Summary updated