Which Tokens, in What Order, and How Fast? A Theoretical Analysis of On-Policy Distillation Objectives

Hubert Strauss, Liam Fowl, Noam RazinarXiv 2026 · arXiv

TL;DR

This paper provides a theoretical deep dive into On-Policy Distillation (OPD), revealing that the choice of objective—sequence-level vs. token-level and the specific divergence used—fundamentally changes what a model learns and how fast. The authors uncover a "future agreement bias" in sequence-level objectives that can lead to undesirable behaviors like over-refusal, and prove that Forward KL is significantly more robust than Reverse KL for teaching models entirely new knowledge.

What Problem Does This Paper Address?

On-Policy Distillation (OPD) is a cornerstone of modern LLM post-training. Unlike traditional distillation where a student mimics a teacher's static dataset, in OPD, the student generates its own text, and the teacher provides feedback on those specific generations. This fixes the "train-test mismatch" but introduces a dizzying array of choices: Do you optimize for the whole sequence at once or token-by-token? Do you use Forward KL (FKL), Reverse KL (RKL), or Jensen-Shannon (JS) divergence?

Until now, researchers mostly relied on "vibes" or the coarse "mode-seeking vs. mode-covering" intuition. This paper moves beyond that, providing a formal theoretical framework to explain why certain objectives lead to repetitive text, slow learning, or "over-refusal" in safety fine-tuning.

Methods

The authors analyze OPD through the lens of the logits gradient—the signal that actually updates the model's internal weights. By decomposing these gradients, they compare:

  1. Granularity: Sequence-level (optimizing the whole path) vs. Token-level (myopic, step-by-step optimization).
  2. Divergence: FKL, RKL, and JS.

They use a tabular student model (where logits are directly trainable) to prove theorems about optimization speed and learning order. They then validate these theoretical "laws" using real-world Transformer models (like Qwen and Gemma) across tasks involving safety, coding, and mathematics.

Datasets

The study employs a mix of synthetic and real-world benchmarks:

  • Binary Tree Navigation: A synthetic task designed to isolate the "future agreement bias."
  • WildJailbreak: Used for safety experiments to measure the balance between helpfulness and harmlessness.
  • New API Code Completion: A custom dataset featuring 96 Python APIs released after the student model's cutoff (Oct 2024 – April 2025) to test "new knowledge" acquisition.
  • MATH Dataset: Used to explore the "capacity gap" between small students and large teachers.

Key Insights and Ideas

  • The Future Agreement Bias: Sequence-level OPD doesn't just care if a token is "right"; it cares if that token leads to a future where the student and teacher will agree. This causes a reverse learning order: the model learns the end of a sentence before the beginning.
  • The Over-Refusal Trap: Because refusals (e.g., "I cannot answer that") are short and formulaic, they have high "future agreement." Sequence-level OPD can accidentally train a model to refuse everything—even benign requests—because it's the easiest way to reach a state of agreement with the teacher.
  • Vanishing Gradients in RKL/JS: The authors prove that for RKL and JS, the gradient can vanish even when the student is completely wrong. If a student is 100% confident in a "bad" token, RKL provides almost no signal to change it. FKL, however, remains well-behaved and forces the model to learn.
  • When RKL Wins: RKL is "mode-seeking." When the student is already competent but the teacher is much larger/smarter (a large capacity gap), RKL helps the student hone in on the teacher's best reasoning patterns rather than trying to cover every possible (and potentially confusing) path the teacher might take.

Key Results

  • Tree Navigation Failure: In a synthetic tree task, sequence-level OPD took exponentially longer to solve the task compared to token-level OPD due to the future agreement bias.
Figure 1: The binary tree navigation task analyzed in Section 3.2 (for depth L = 3). The student starts at the root, and its goal is to produce with high probability the path y⋆= (A, A, . . . , A), which reaches a fixed target leaf. The teacher assigns high probability to choosing the left child at the root and every node in its left subtree (δ = exp(−L)), but chooses uniformly among children if the initial token is incorrect.
  • Safety Accuracy: On the WildJailbreak dataset, sequence-level OPD caused a Qwen student to refuse nearly all benign requests, even though the teacher only refused 17.5%. Token-level objectives avoided this entirely.
Figure 3: Future agreement bias leads sequence-level OPD to cause over-refusal of benign requests. We trained a Qwen3-1.7B student using sequence-level and token-level OPD objectives with a Qwen3-8B teacher. The training prompts were taken from the WildJailbreak dataset and consist of an equal mixture of harmful and benign requests. Reported values are computed over training prompts and averaged over three seeds, with shaded areas marking standard deviation (results on a test set showed identical trends). Notice that, although the teacher refuses only 17.5% of the benign requests, sequence-level OPD causes the student to refuse nearly all of them due to the future agreement bias characterized in Section 3.1. Consequently, sequence-level OPD lowers the safety accuracy of the student (i.e., the percentage of prompts for which it correctly refuses a harmful request or answers a benign one), whereas token-level OPD substantially improves it by lowering the refusal rate on benign requests and raising the refusal rate on harmful requests.
  • Learning New Knowledge: In the code completion task, FKL (and hybrid mixtures) successfully taught the model new APIs. RKL and JS struggled significantly because the student initially assigned these new tokens near-zero probability, leading to the vanishing gradient problem.

Limitations and Future Work

The authors acknowledge that their primary proofs rely on a "tabular student" (directly optimizing logits). While their experiments show these insights carry over to Transformers, the complex interaction between neural network architecture and these objectives remains an open area for study.

Additionally, while they mostly highlight the harms of future agreement bias, they briefly show in Appendix A that it can be beneficial if a task has multiple correct paths and one is much easier for the student to "agree" on. Identifying more scenarios where sequence-level OPD outperforms token-level is a key direction for future research.

Summary updated

Abstract

On-policy distillation (OPD) trains a student language model to match a teacher model based on its own generations. When using OPD, one must choose whether to apply it at the sequence or token level and which divergence to minimize. However, despite the wide adoption of OPD in practice, the formal understanding of which OPD objective is appropriate for a given setting is limited. Toward addressing this gap, we theoretically analyze how the choice of OPD objective governs which tokens are prioritized, in what order, and how efficiently. Our analysis reveals that sequence-level OPD exhibits a future agreement bias, which myopic token-level objectives avoid. This bias induces a reverse learning order, in which sequence-level OPD first matches the teacher at later token positions, while at earlier positions it can favor tokens that are improbable under the teacher, and thus often undesirable. As for the choice of divergence, we prove that it strongly influences the amenability of OPD to gradient-based optimization. In particular, while the forward KL divergence enjoys a well-behaved objective landscape, other $f$-divergences, such as reverse KL, can suffer from vanishing gradients even when the student and teacher are far apart. Across language model post-training settings, we demonstrate the implications of these theoretical findings for which OPD objective is preferable, highlighting the importance of accounting for the interplay between the student, teacher, and task at hand.
BibTeX
@misc{strauss2026which,
  author = {Hubert Strauss and Liam Fowl and Noam Razin},
  title = {{Which Tokens, in What Order, and How Fast? A Theoretical Analysis of On-Policy Distillation Objectives}},
  year = {2026},
  eprint = {2611.23397},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23397},
}
Figure