ToDi: Token-wise Distillation via Fine-Grained Divergence Control

Soyoung Jung, Suwan Yoon, DongGeon Kim, Hwanhee Lee2025 · DOI

TL;DR

ToDi (Token-wise Distillation) moves away from "one-size-fits-all" distillation losses by dynamically blending Forward and Reverse KL divergence for every single token in a model's vocabulary. By using a fine-grained, sigmoid-based weighting function, ToDi ensures that student models receive the exact corrective signal they need—whether that's "pushing up" an underestimated token or "pulling down" an overestimated one—leading to state-of-the-art performance in LLM compression.

What Problem Does This Paper Address?

Knowledge Distillation (KD) is the go-to method for shrinking massive Large Language Models (LLMs) into smaller, deployable versions. Traditionally, researchers choose a specific "divergence" (a mathematical way to measure the difference between the teacher's and student's predictions) and apply it uniformly across the entire vocabulary.

The two heavy hitters are Forward KL (FKL) and Reverse KL (RKL). FKL tends to make the student "mean-seeking" (covering all the teacher's modes but potentially becoming blurry), while RKL makes it "mode-seeking" (focusing on the highest peaks but risking mode collapse).

The authors argue that applying these losses uniformly is sub-optimal. At any given moment, a student model might be underestimating some tokens while overestimating others. A global loss function can't provide the specific "nudge" needed for individual tokens, leading to a student that never quite captures the fine-grained nuances of the teacher.

Methods

The authors propose ToDi, which introduces Fine-Grained Divergence Control.

  1. Gradient Analysis: The researchers first proved mathematically that FKL and RKL play complementary roles. FKL provides a strong "push-up" signal for tokens the student underestimates, while RKL provides a strong "pull-down" signal for tokens the student overestimates.
  2. Token-wise Weighting: Instead of picking one loss, ToDi calculates a weight ($\alpha$) for every token $i$ at every time step $t$. This weight is determined by a sigmoid function of the log-ratio between the teacher’s probability ($p$) and the student’s probability ($q$): $$\alpha_{t,i} = \sigma\left(\beta \cdot \log \frac{p(v_i)}{q(v_i)}\right)$$
  3. Dynamic Blending: If the student is too low on a token ($p > q$), $\alpha$ increases, giving more weight to FKL to "push" the probability up. If the student is too high ($q > p$), $\alpha$ decreases, favoring RKL to "pull" it down.
  4. Stop-Gradient Trick: To keep training stable, the authors apply a "stop-gradient" to the weighting function itself. This ensures the model optimizes the distribution based on the weights without the weights themselves creating a chaotic feedback loop during backpropagation.

Datasets

The models were trained on the databricks/dolly-15k dataset (11k training, 1k validation, 500 test samples).

Evaluation was performed across five diverse instruction-following benchmarks to ensure robustness:

  • DollyEval: General instruction following.
  • S-NI (Super-NaturalInstructions): Diverse NLP tasks.
  • UnNI (Unnatural Instructions): Synthetically generated instructions.
  • SelfInst: User-oriented instructions.
  • VicunaEval: Challenging prompts for chatbot evaluation.

Key Insights and Ideas

  • The "Push-Pull" Duality: The most elegant insight is the realization that FKL and RKL aren't just different formulas; they are directional tools. FKL is the accelerator for underestimated tokens; RKL is the brake for overestimated ones.
  • Granularity Matters: Previous "adaptive" methods adjusted weights at the sequence level or for the whole vocabulary at once. ToDi proves that the real "action" happens at the individual token level.
  • Efficiency without Sorting: Unlike some recent adaptive methods (like AKL) that require sorting the entire vocabulary (an $O(V \log V)$ operation), ToDi is $O(V)$. It calculates weights via simple element-wise operations, making it as computationally efficient as standard KL distillation.

Key Results

ToDi was tested using several teacher-student pairs, including GPT2-1.5B → GPT2-120M and LLaMA2-7B → TinyLLaMA-1.1B.

  • ROUGE-L Superiority: ToDi consistently achieved the highest average scores across all five benchmarks. For the LLaMA2 → TinyLLaMA setup, ToDi reached an average ROUGE-L of 24.83, outperforming FKL (24.00) and RKL (24.09).
  • GPT-4 Evaluation: In pairwise "blind" tests where GPT-4 judged which model gave a better response, ToDi-trained models won significantly more often than those trained with standard FKL, RKL, or even more complex adaptive baselines.
  • Stability: Training curves showed that ToDi provides a smoother, more stable convergence compared to methods that use abrupt "step functions" for weighting.

Limitations and Future Work

  • Vocabulary Alignment: ToDi currently requires the teacher and student to share the exact same vocabulary. This is a common constraint in KD, but it limits the ability to distill knowledge between models from different families (e.g., distilling GPT-4 into a Llama-based student).
  • Logit Access: The method requires access to the full probability distribution (logits) of the teacher. This makes it difficult to use with "closed" models (like the GPT-4 API) that only provide the top few tokens or just text.
  • Scaling: While tested on models up to 7B parameters, the authors note that testing on "extremely large-scale" models (70B+) was not possible due to hardware constraints. Future work could explore if these token-wise benefits scale linearly with model size.

Summary updated

Abstract

Large language models (LLMs) offer impressive performance but are impractical for resource-constrained deployment due to high latency and energy consumption.Knowledge distillation (KD) addresses this by transferring knowledge from a large teacher to a smaller student model.However, conventional KD, notably approaches like Forward KL (FKL) and Reverse KL (RKL), apply uniform divergence loss across the entire vocabulary, neglecting token-level prediction discrepancies.By investigating these representative divergences via gradient analysis, we reveal that FKL boosts underestimated tokens, while RKL suppresses overestimated ones, showing their complementary roles.Based on this observation, we propose Token-wise Distillation (ToDi), a novel method that adaptively combines FKL and RKL per token using a sigmoid-based weighting function derived from the teacherstudent probability log-ratio.ToDi dynamically emphasizes the appropriate divergence for each token, enabling precise distribution alignment.We demonstrate that ToDi consistently outperforms recent distillation baselines using uniform or less granular strategies across instruction-following benchmarks.Extensive ablation studies and efficiency analysis further validate ToDi's effectiveness and practicality.
BibTeX
@misc{jung2025todi,
  author = {Soyoung Jung and Suwan Yoon and DongGeon Kim and Hwanhee Lee},
  title = {{ToDi: Token-wise Distillation via Fine-Grained Divergence Control}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.409},
}
Figure