ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence

Guanghui Wang, Zhiyong Yang, Zitai Wang, Shi Wang, Qianqian Xu, Qingming Huang

This paper tackles a fundamental tension in Knowledge Distillation (KD): how to make a student model learn the "richness" of a teacher's distribution without becoming too blurry or too narrow. The authors introduce ABKD, a framework that uses $\alpha$-$\beta$-divergence to precisely control the "concentration" of probability mass during training.

TL;DR

ABKD moves beyond the rigid choice of Forward or Reverse KL divergence in knowledge distillation by using a generalized $\alpha$-$\beta$-divergence. By decoupling how a student focuses on difficult samples versus its own confident predictions, ABKD achieves superior performance across 17 datasets in vision and language without adding any training overhead or extra parameters.

What Problem Does This Paper Address?

In Knowledge Distillation, a small "student" model learns to mimic the output probabilities of a large "teacher." Traditionally, this is done using Forward KL Divergence (FKLD). However, FKLD is "mode-covering"—it forces the student to spread its probability mass across the teacher's entire distribution, often resulting in a "blurry" student that fails to prioritize the correct target class.

Probability mass distributions comparing FKLD, RKLD, and ABKD

Recently, researchers tried Reverse KL Divergence (RKLD), which is "mode-seeking." While this helps the student focus on the most likely classes, it often overcorrects, causing the student to ignore the teacher's nuanced "soft labels" and collapse into a simple one-hot-like predictor.

The authors identify that the core issue is an imbalance between two effects:

  1. Hardness-Concentration: Focusing on classes where the error between teacher and student is large.
  2. Confidence-Concentration: Focusing on classes where the student is already highly confident.

Standard methods entangle these two effects in extreme, often suboptimal ways.

Methods

The authors propose ABKD (Alpha-Beta Knowledge Distillation). Instead of standard KL, they employ the $\alpha$-$\beta$-divergence, a power-form divergence that unifies many common measures (KL, Reverse KL, Hellinger distance, etc.) into a single searchable space.

The key technical contribution is an analysis of the Log Mass Ratio. By tracking how probability mass is reassigned during a single gradient update, they show exactly how the hyperparameters $\alpha$ and $\beta$ influence the student:

  • $\alpha$ (Hardness): Controls how aggressively the model penalizes errors. A smaller $\alpha$ amplifies the focus on "hard" classes.
  • $\beta$ (Confidence): Controls how much the model weights the student's own confidence. A larger $\beta$ focuses the learning on the student's most certain predictions.

Crucially, ABKD is a "plug-and-play" loss function. It requires no extra trainable parameters and maintains the same training speed as vanilla KD.

Datasets

The authors validated ABKD on a massive scale across 17 datasets and 12 teacher-student configurations:

  • NLP (Instruction-Following): databricks-dolly-15k, Self-Instruct, Vicuna, Super-Natural Instructions, and Unnatural Instruction.
  • Vision (Image Classification): CIFAR-100, ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, UCF101, DTD, and EuroSAT.
  • Models: They tested everything from GPT-2 and LLaMA2 (7B to 3B) in language to ResNet, VGG, and WideResNet in vision.

Key Insights and Ideas

  • The "Goldilocks" Divergence: The paper proves that FKLD and RKLD are just two extreme points in the $\alpha$-$\beta$ plane. ABKD allows practitioners to find the "just right" middle ground that balances mode-covering and mode-seeking behaviors.
  • Theoretical Grounding: The authors provide a formal proof showing that standard $\alpha$-divergence (where $\alpha + \beta = 1$) is too restrictive because it couples the two concentration effects. By allowing $\alpha$ and $\beta$ to vary independently, ABKD gains the flexibility needed for complex tasks.
  • Efficiency over SGOs: Many recent SOTA distillation methods rely on "Student-Generated Outputs" (SGOs), which require sampling during training and can be 7x slower. ABKD achieves better results than these methods while remaining as fast as the simplest baseline.
Training speed comparison showing ABKD's efficiency

Key Results

  • NLP Gains: On instruction-following tasks, ABKD improved ROUGE-L scores by 0.81 to 3.31 points over standard FKLD and RKLD baselines.
  • Vision Consistency: In vision tasks (e.g., ResNet110 to ResNet20), ABKD consistently outperformed specialized KD methods like DKD and TTM.
  • Base-to-New Generalization: In "zero-shot" style settings where a model is trained on base classes and tested on new ones, ABKD showed significantly better generalization than standard CLIP-based fine-tuning.
  • Scaling: The benefits of ABKD actually increased as the teacher model got larger, suggesting it effectively bridges the "capacity gap" between massive teachers and tiny students.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the authors provide clear guidelines (e.g., use larger $\beta$ for high-dimensional outputs like text), $\alpha$ and $\beta$ still need to be tuned for optimal performance on new architectures.
  • Static vs. Dynamic: Currently, $\alpha$ and $\beta$ are fixed throughout training. The authors suggest that future work could explore dynamic schedules—perhaps starting with a mode-covering $\alpha$ to explore the distribution and shifting to a mode-seeking $\alpha$ to sharpen the final model.

Summary updated

Abstract

Knowledge Distillation (KD) transfers knowledge from a large teacher model to a smaller student model by minimizing the divergence between their output distributions, typically using forward Kullback-Leibler divergence (FKLD) or reverse KLD (RKLD). It has become an effective training paradigm due to the broader supervision information provided by the teacher distribution compared to one-hot labels. We identify that the core challenge in KD lies in balancing two mode-concentration effects: the \textbf{\textit{Hardness-Concentration}} effect, which refers to focusing on modes with large errors, and the \textbf{\textit{Confidence-Concentration}} effect, which refers to focusing on modes with high student confidence. Through an analysis of how probabilities are reassigned during gradient updates, we observe that these two effects are entangled in FKLD and RKLD, but in extreme forms. Specifically, both are too weak in FKLD, causing the student to fail to concentrate on the target class. In contrast, both are too strong in RKLD, causing the student to overly emphasize the target class while ignoring the broader distributional information from the teacher. To address this imbalance, we propose ABKD, a generic framework with $\alpha$-$\beta$-divergence. Our theoretical results show that ABKD offers a smooth interpolation between FKLD and RKLD, achieving a better trade-off between these effects. Extensive experiments on 17 language/vision datasets with 12 teacher-student settings confirm its efficacy.
BibTeX
@inproceedings{wang2025abkd,
  author = {Guanghui Wang and Zhiyong Yang and Zitai Wang and Shi Wang and Qianqian Xu and Qingming Huang},
  title = {{ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\textbackslash\{\}alpha$-$\textbackslash\{\}beta$-Divergence}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
}
Figure