TL;DR
On-policy distillation (OPD) is a powerful tool for transferring reasoning capabilities from large teacher models to smaller student LLMs, but standard token-level implementations are notoriously brittle on long sequences. This paper identifies key empirical failure modes—such as highly imbalanced supervision and tokenizer mismatches—and introduces Teacher Top-K Local Support Matching (LSM). This truncated reverse-KL objective stabilizes training and delivers up to a +19.8% performance boost across reasoning and agentic tasks.
What Problem Does This Paper Address?
When post-training Large Language Models (LLMs) for complex reasoning or agentic tasks, developers frequently turn to On-Policy Distillation (OPD). By evaluating the student model's own generated rollouts (on-policy trajectories) using a stronger teacher model, OPD provides dense, token-level feedback.
However, the standard way to implement this is through sampled-token comparison, which drives updates based on the teacher-student log-probability difference of just the single sampled token. On long rollouts, this signal quickly degrades. The student's prefix drifts away from the teacher's typical distribution, causing the teacher's guidance to become unreliable, leading to issues like repetition loops, entropy collapse, or severe penalties due to minor tokenizer mismatches. This paper investigates the fundamental bias-variance trade-offs of this setup and fixes its core vulnerabilities.
Methods
The authors approach the problem by first analyzing the theoretical trade-offs between sequence-level and token-level objectives, and then designing a more robust local objective:
- Theoretical Variance Analysis: They prove that while sequence-level reverse-KL minimization is unbiased, its worst-case variance scales quadratically worse with sequence length ($O(T^4)$) compared to token-level estimators ($O(T^2)$). This justifies keeping the supervision strictly token-level to control variance in long-horizon tasks.
- Teacher Top-K Local Support Matching (LSM): Instead of evaluating only the single sampled token, LSM compares the student and teacher distributions over a local support set containing the $K$ highest-probability tokens under the teacher at that specific prefix.
- Support-Set Renormalization: Within this top-K support set, both the teacher and student distributions are renormalized before computing a truncated reverse-KL divergence. This ensures that gradients do not propagate to tokens outside the support set, preventing optimization instability.
- Rollout Stabilization: The method integrates top-$p$ rollout sampling to keep student generations closer to typical continuations, alongside special-token masking to prevent artificial penalties arising from mismatched tokenizer conventions.
Datasets
The authors evaluated their method across single-task reasoning and multi-task agentic settings using the following datasets:
- Math Reasoning: Trained on the English portion of DAPO-Math-17K. Evaluated using pass@1 and avg@32 metrics on Math500, AIME24, AIME25, Minerva, and OlympiadBench.
- Agentic Tasks: ALFWorld (a multi-turn interactive text environment) and WebShop (used to verify single-task agentic transfer).
Key Insights and Ideas
- The Causal Reward-to-Go Variance Trap: Coupling token updates to all future rewards along a trajectory dramatically inflates gradient variance. Token-level supervision is mathematically necessary to keep long-sequence training stable.
- Imbalanced Penalization: Standard sampled-token OPD is heavily biased toward penalizing the student's current token rather than offering constructive reinforcement, as the vast majority of sampled tokens receive negative local rewards.
- The Tokenizer Tax: If a teacher and student model segment the exact same raw text differently (e.g.,
<think>split into different sub-tokens), a single-token comparison mistakes this tokenizer mismatch for a semantic error, distorting the reward signal. - Renormalization is Non-Negotiable: Truncating the distribution without renormalizing the remaining probabilities causes rapid policy collapse. Renormalization keeps the student focused on the teacher's preferred local options.
Key Results
- Single-Task Math Improvements: On single-task math reasoning, LSM raised the average score of a Qwen2.5-7B-Instruct student from a baseline of 28.2 to 41.7, substantially outperforming standard sampled-token OPD (36.4).
- Multi-Task Gains: In an alternating multi-task setup mixing math and agentic tasks, LSM achieved a +19.8% performance gain in math reasoning (improving the average score from 34.8 to 41.7) while maintaining a highly competitive 95.3% success rate on ALFWorld.
- Robustness to Mismatch: While special-token masking significantly helped the standard baseline (improving it from 36.4 to 40.7), LSM proved inherently robust to tokenizer discrepancies, showing stable performance with or without explicit masking.
Limitations and Future Work
- Training-Inference Mismatch: The prefixes are generated under a rollout policy (like top-$p$ sampling), but the training engine updates the model without mathematically correcting for this sampling process.
- Imperfect Success Proxy: Local teacher matching can occasionally fall prey to reward-hacking. For instance, if a student drifts into an unproductive loop, the teacher might still assign high conditional probabilities to generic reasoning fillers, encouraging the student to over-continue rather than terminate cleanly.
- Future Directions: The authors suggest exploring adaptive top-$p$ truncation for the KL support set rather than a fixed top-$K$, and setting up a rigorous matched-compute comparison between OPD and Reinforcement Learning (RL) to better understand multi-task transfer dynamics.
Summary updated