Mode Seeking Distillation via Reverse KL Divergence

28 concepts19 papers1 proposal

When compressing a large generative model into a smaller one, the standard approach has long been to minimize the forward Kullback-Leibler (KL) divergence between the teacher's distribution $P$ and the student's distribution $Q$:

$$D_{\text{KL}}(P \parallel Q) = \sum_{x} P(x) \log \frac{P(x)}{Q(x)}$$

Because the target distribution $P(x)$ sits in the numerator of the logarithm, the student model is heavily penalized if it assigns a low probability $Q(x)$ to any region where the teacher $P(x)$ has mass. This mathematical property forces the student to be mode-covering (or mean-seeking). The student attempts to stretch its probability mass across the entire support of the teacher's distribution.

However, when there is a severe capacity mismatch between a massive teacher and a compact student, this mode-covering behavior breaks down.

The Capacity Mismatch Problem

A low-capacity student simply does not have the parameters required to represent the full, complex, multi-modal distribution of a large language model or diffusion model. Forced by the forward KL divergence to cover everything, the student ends up spreading its limited probability mass too thin.

As Yuxian Gu et al. observe in the context of large language models, this forces the student to assign unnaturally high probabilities to low-quality regions or "dead zones" of the search space. Instead of mastering a subset of the data, the student becomes a master of none, leading to blurred outputs, structural errors, and frequent hallucinations. Rishabh Agarwal demonstrates that when a student cannot capture the full complexity of the teacher, a mode-covering objective causes it to fail to represent any part of the distribution accurately.

Prioritizing Precision via Reverse KL

To resolve this, a growing body of research flips the objective function, utilizing the reverse KL divergence instead:

$$D_{\text{KL}}(Q \parallel P) = \sum_{x} Q(x) \log \frac{Q(x)}{P(x)}$$

In this formulation, the student's distribution $Q(x)$ weights the objective. If the teacher assigns zero or low probability to a region ($P(x) \approx 0$), the student is heavily penalized if it places any mass there ($Q(x) > 0$). Crucially, if the teacher has a high-probability mode that the student chooses to ignore ($Q(x) = 0$ while $P(x) > 0$), the corresponding term in the sum drops to zero.

This creates a mode-seeking behavior. Rather than attempting to cover every single mode of the teacher, the student focuses its limited capacity entirely on the teacher's highest-confidence peaks.

As explained by Sungmin Cha et al., this shifts the distillation paradigm from maximizing recall (covering the entire distribution) to maximizing precision (ensuring that whatever the student generates is highly accurate).

This precision-first behavior is highly advantageous across several domains:

  • Structured Output and Code Generation: In tasks like Text-to-SQL, averaging across multiple valid syntactic paths results in broken, uncompilable code. Qihuang Zhong et al. show that mode-seeking objectives force the student to commit to a single, high-confidence logical trajectory, preserving syntactic precision.
  • Information Retrieval: For high-stakes retrieval and query rewriting, Duy A. Nguyen et al. demonstrate that reverse KL prevents the student from wasting its capacity on irrelevant or noisy tail outputs, significantly boosting precision.
  • Reasoning Chains: When distilling multi-step reasoners, Daniele Paliotta et al. show that forcing the student to prioritize the most probable tokens helps it maintain coherent, high-confidence logical paths.

Stabilizing the Mode-Seeking Gradients

While reverse KL offers clear conceptual advantages for precision, optimizing it directly can be unstable. In high-cardinality output spaces—such as vocabulary distributions containing tens of thousands of tokens—the gradients can become highly volatile.

Max Ryabinin et al. identify that when distilling over large output spaces, the long tail of low-probability classes can produce noisy gradients that overwhelm the training signal. They introduce a proxy target distribution to stabilize the ratio of gradients between high- and low-probability classes, allowing reverse KL to scale effectively to large datasets.

Other approaches modify the teacher's distribution before distillation begins. For example, Doohyuk Jang et al. use parameter-efficient prompt tuning to dynamically adapt the teacher's output space to the student's capacity, making the primary modes easier for the student to locate and track.

The Diversity Dilemma

Pure mode-seeking behavior is not without its drawbacks. By forcing the student to focus exclusively on the primary peaks of the teacher's distribution, the model risks severe mode collapse, where it completely loses the ability to generate diverse outputs.

Chin-Wei Huang et al. warn that unconstrained probability distillation via reverse KL frequently introduces audio or visual artifacts due to this collapse. Furthermore, dropping the entropy of the distribution can be catastrophic if the student model is intended for downstream reinforcement learning (RL). As Jiliang Ni et al. note, strict mode-seeking behavior suppresses the token-level entropy required for effective RL exploration, making a modified forward KL with tail suppression preferable in exploration-heavy pipelines. Similarly, in fast visual synthesis, Ruibin Li et al. show that distribution matching distillation suffers from mode collapse unless the mode-seeking objective is strictly isolated to late-stage image refinement steps.

To navigate this tension between precision and diversity, recent frameworks seek a middle ground through generalized or symmetric divergences. Guanghui Wang 0001 et al. introduce an $\alpha$-$\beta$-divergence framework that allows practitioners to smoothly interpolate between mode-covering and mode-seeking behaviors via hyperparameter tuning. Alternatively, Yuqiao Wen et al. employ symmetric measures like the Jensen-Shannon divergence. By blending both forward and reverse characteristics, these hybrid objectives prevent the student from producing overly generic, blurry outputs while ensuring it does not entirely drop critical, high-probability alternative modes.

Key Foundational Work

Probability Distillation: A Caveat and Alternatives (2019) — contributed 1 concept elements to this direction.
Scaling Ensemble Distribution Distillation to Many Classes with Proxy Targets (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_mode_seeking_distillation_abbe7e88,
  title = {mode-seeking distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mode-seeking-distillation/rdr_abbe7e8822b945bcb3151b23853f1edb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_mode_seeking_distillation_abbe7e88_v1,
  title = {mode-seeking distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mode-seeking-distillation/rdr_abbe7e8822b945bcb3151b23853f1edb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake