TL;DR
This paper provides a theoretical deep dive into On-Policy Distillation (OPD), revealing that the choice of objective—sequence-level vs. token-level and the specific divergence used—fundamentally changes what a model learns and how fast. The authors uncover a "future agreement bias" in sequence-level objectives that can lead to undesirable behaviors like over-refusal, and prove that Forward KL is significantly more robust than Reverse KL for teaching models entirely new knowledge.
What Problem Does This Paper Address?
On-Policy Distillation (OPD) is a cornerstone of modern LLM post-training. Unlike traditional distillation where a student mimics a teacher's static dataset, in OPD, the student generates its own text, and the teacher provides feedback on those specific generations. This fixes the "train-test mismatch" but introduces a dizzying array of choices: Do you optimize for the whole sequence at once or token-by-token? Do you use Forward KL (FKL), Reverse KL (RKL), or Jensen-Shannon (JS) divergence?
Until now, researchers mostly relied on "vibes" or the coarse "mode-seeking vs. mode-covering" intuition. This paper moves beyond that, providing a formal theoretical framework to explain why certain objectives lead to repetitive text, slow learning, or "over-refusal" in safety fine-tuning.
Methods
The authors analyze OPD through the lens of the logits gradient—the signal that actually updates the model's internal weights. By decomposing these gradients, they compare:
- Granularity: Sequence-level (optimizing the whole path) vs. Token-level (myopic, step-by-step optimization).
- Divergence: FKL, RKL, and JS.
They use a tabular student model (where logits are directly trainable) to prove theorems about optimization speed and learning order. They then validate these theoretical "laws" using real-world Transformer models (like Qwen and Gemma) across tasks involving safety, coding, and mathematics.
Datasets
The study employs a mix of synthetic and real-world benchmarks:
- Binary Tree Navigation: A synthetic task designed to isolate the "future agreement bias."
- WildJailbreak: Used for safety experiments to measure the balance between helpfulness and harmlessness.
- New API Code Completion: A custom dataset featuring 96 Python APIs released after the student model's cutoff (Oct 2024 – April 2025) to test "new knowledge" acquisition.
- MATH Dataset: Used to explore the "capacity gap" between small students and large teachers.
Key Insights and Ideas
- The Future Agreement Bias: Sequence-level OPD doesn't just care if a token is "right"; it cares if that token leads to a future where the student and teacher will agree. This causes a reverse learning order: the model learns the end of a sentence before the beginning.
- The Over-Refusal Trap: Because refusals (e.g., "I cannot answer that") are short and formulaic, they have high "future agreement." Sequence-level OPD can accidentally train a model to refuse everything—even benign requests—because it's the easiest way to reach a state of agreement with the teacher.
- Vanishing Gradients in RKL/JS: The authors prove that for RKL and JS, the gradient can vanish even when the student is completely wrong. If a student is 100% confident in a "bad" token, RKL provides almost no signal to change it. FKL, however, remains well-behaved and forces the model to learn.
- When RKL Wins: RKL is "mode-seeking." When the student is already competent but the teacher is much larger/smarter (a large capacity gap), RKL helps the student hone in on the teacher's best reasoning patterns rather than trying to cover every possible (and potentially confusing) path the teacher might take.
Key Results
- Tree Navigation Failure: In a synthetic tree task, sequence-level OPD took exponentially longer to solve the task compared to token-level OPD due to the future agreement bias.
- Safety Accuracy: On the WildJailbreak dataset, sequence-level OPD caused a Qwen student to refuse nearly all benign requests, even though the teacher only refused 17.5%. Token-level objectives avoided this entirely.
- Learning New Knowledge: In the code completion task, FKL (and hybrid mixtures) successfully taught the model new APIs. RKL and JS struggled significantly because the student initially assigned these new tokens near-zero probability, leading to the vanishing gradient problem.
Limitations and Future Work
The authors acknowledge that their primary proofs rely on a "tabular student" (directly optimizing logits). While their experiments show these insights carry over to Transformers, the complex interaction between neural network architecture and these objectives remains an open area for study.
Additionally, while they mostly highlight the harms of future agreement bias, they briefly show in Appendix A that it can be beneficial if a task has multiple correct paths and one is much easier for the student to "agree" on. Identifying more scenarios where sequence-level OPD outperforms token-level is a key direction for future research.
Summary updated