Abstract
Knowledge Distillation (KD) typically relies on Forward or Reverse KL Divergence, both of which struggle to properly allocate probability mass due to extreme "mode-concentration" behaviors. This paper introduces ABKD, a novel framework using $\alpha$-$\beta$-divergence to smoothly interpolate between these extremes, allowing for a tunable balance between focusing on hard examples and confident predictions. The result is a drop-in replacement loss function that boosts performance across a wide range of vision and language tasks without adding training overhead.
Introduction
Knowledge Distillation is a popular technique where a smaller "student" model is trained to mimic the output distribution of a larger, smarter "teacher" model. To do this, researchers need a mathematical way to measure the difference (divergence) between the two models' predictions.
Historically, the go-to choices are Forward Kullback-Leibler Divergence (FKLD) or Reverse KLD (RKLD). However, both have fatal flaws:
- FKLD is too "smooth." It treats all mismatches equally, causing the student to spread its guesses too broadly and fail to confidently predict the correct target class.
- RKLD is too "sharp." It forces the student to focus almost exclusively on the target class, effectively ignoring the rich, nuanced "soft label" information that makes the teacher's distribution so valuable in the first place.
The authors frame this problem through two novel concepts: Hardness-Concentration (focusing on classes with large errors) and Confidence-Concentration (focusing on classes where the student is already confident). FKLD has too little of both, while RKLD has too much of both. The paper addresses the lack of a systematic, mathematically grounded way to balance these two effects.
Methods
To fix the imbalance of FKLD and RKLD, the authors propose ABKD (Alpha-Beta Knowledge Distillation). Instead of standard KL divergence, ABKD uses the $\alpha$-$\beta$-divergence, a generalized family of divergence functions. By tuning two hyperparameters, $\alpha$ and $\beta$, this function can perfectly recreate FKLD ($\alpha=1, \beta=0$) and RKLD ($\alpha=0, \beta=1$), as well as everything in between.
- $\alpha$ controls Hardness-Concentration: A smaller $\alpha$ makes the model more aggressively penalize errors on "hard" classes (where the teacher and student disagree heavily).
- $\beta$ controls Confidence-Concentration: A larger $\beta$ forces the model to focus its error-correction on classes where the student already has high confidence, sharpening the final output.
By decoupling these effects, ABKD allows the student to learn the broad distributional knowledge of the teacher while still maintaining a sharp, accurate focus on the target class. Because ABKD only changes the mathematical objective (the loss function), it requires no extra trainable parameters and adds zero computational overhead during training.
Experimental Results
The authors evaluated ABKD extensively across 17 datasets covering NLP and Computer Vision.
NLP and Instruction Following
When distilling GPT-2 XL (1.5B) into a smaller GPT-2 (0.1B), simply swapping the loss function to ABKD yielded performance improvements of 0.81 to 3.31 ROUGE-L points over standard FKLD and RKLD across five datasets (Dolly Eval, Self-Instruct, etc.).
Computer Vision and Generalization
In a base-to-new class generalization setup using CLIP (ViT-L/14 to ViT-B/16), ABKD achieved an average Harmonic Mean accuracy of 84.17% across 11 datasets (CIFAR-100, ImageNet, etc.), outperforming standard KD (83.63%) and other state-of-the-art distillation methods.
Efficiency and Compatibility
Because ABKD is just a loss function modification, it matches the training speed of vanilla KD. In contrast, other modern LLM distillation methods that rely on generating text during training take 1.6x to 7x longer to train. Furthermore, ABKD can be combined with existing state-of-the-art KD methods (like DKD or TTM) to further boost their performance.
Discussion and Conclusion
The $\alpha$-$\beta$-divergence works because it fundamentally alters the gradient weighting, allowing for a "just right" allocation of probability mass. The authors discovered a principled guideline for tuning: low-dimensional tasks (e.g., CIFAR-100) benefit from large $\alpha$ and small $\beta$, while high-dimensional tasks (e.g., LLMs) require smaller $\alpha$ and larger $\beta$ to avoid local optima.
While ABKD introduces two new hyperparameters, the performance gains and zero-overhead training make it a powerful drop-in replacement for standard KL divergence in knowledge distillation. Future work may explore applying these insights to preference alignment and reinforcement learning paradigms.
Summary updated