This paper tackles a fundamental tension in Knowledge Distillation (KD): how to make a student model learn the "richness" of a teacher's distribution without becoming too blurry or too narrow. The authors introduce ABKD, a framework that uses $\alpha$-$\beta$-divergence to precisely control the "concentration" of probability mass during training.
TL;DR
ABKD moves beyond the rigid choice of Forward or Reverse KL divergence in knowledge distillation by using a generalized $\alpha$-$\beta$-divergence. By decoupling how a student focuses on difficult samples versus its own confident predictions, ABKD achieves superior performance across 17 datasets in vision and language without adding any training overhead or extra parameters.
What Problem Does This Paper Address?
In Knowledge Distillation, a small "student" model learns to mimic the output probabilities of a large "teacher." Traditionally, this is done using Forward KL Divergence (FKLD). However, FKLD is "mode-covering"—it forces the student to spread its probability mass across the teacher's entire distribution, often resulting in a "blurry" student that fails to prioritize the correct target class.
Recently, researchers tried Reverse KL Divergence (RKLD), which is "mode-seeking." While this helps the student focus on the most likely classes, it often overcorrects, causing the student to ignore the teacher's nuanced "soft labels" and collapse into a simple one-hot-like predictor.
The authors identify that the core issue is an imbalance between two effects:
- Hardness-Concentration: Focusing on classes where the error between teacher and student is large.
- Confidence-Concentration: Focusing on classes where the student is already highly confident.
Standard methods entangle these two effects in extreme, often suboptimal ways.
Methods
The authors propose ABKD (Alpha-Beta Knowledge Distillation). Instead of standard KL, they employ the $\alpha$-$\beta$-divergence, a power-form divergence that unifies many common measures (KL, Reverse KL, Hellinger distance, etc.) into a single searchable space.
The key technical contribution is an analysis of the Log Mass Ratio. By tracking how probability mass is reassigned during a single gradient update, they show exactly how the hyperparameters $\alpha$ and $\beta$ influence the student:
- $\alpha$ (Hardness): Controls how aggressively the model penalizes errors. A smaller $\alpha$ amplifies the focus on "hard" classes.
- $\beta$ (Confidence): Controls how much the model weights the student's own confidence. A larger $\beta$ focuses the learning on the student's most certain predictions.
Crucially, ABKD is a "plug-and-play" loss function. It requires no extra trainable parameters and maintains the same training speed as vanilla KD.
Datasets
The authors validated ABKD on a massive scale across 17 datasets and 12 teacher-student configurations:
- NLP (Instruction-Following):
databricks-dolly-15k,Self-Instruct,Vicuna,Super-Natural Instructions, andUnnatural Instruction. - Vision (Image Classification):
CIFAR-100,ImageNet,Caltech101,OxfordPets,StanfordCars,Flowers102,Food101,FGVCAircraft,SUN397,UCF101,DTD, andEuroSAT. - Models: They tested everything from GPT-2 and LLaMA2 (7B to 3B) in language to ResNet, VGG, and WideResNet in vision.
Key Insights and Ideas
- The "Goldilocks" Divergence: The paper proves that FKLD and RKLD are just two extreme points in the $\alpha$-$\beta$ plane. ABKD allows practitioners to find the "just right" middle ground that balances mode-covering and mode-seeking behaviors.
- Theoretical Grounding: The authors provide a formal proof showing that standard $\alpha$-divergence (where $\alpha + \beta = 1$) is too restrictive because it couples the two concentration effects. By allowing $\alpha$ and $\beta$ to vary independently, ABKD gains the flexibility needed for complex tasks.
- Efficiency over SGOs: Many recent SOTA distillation methods rely on "Student-Generated Outputs" (SGOs), which require sampling during training and can be 7x slower. ABKD achieves better results than these methods while remaining as fast as the simplest baseline.
Key Results
- NLP Gains: On instruction-following tasks, ABKD improved ROUGE-L scores by 0.81 to 3.31 points over standard FKLD and RKLD baselines.
- Vision Consistency: In vision tasks (e.g., ResNet110 to ResNet20), ABKD consistently outperformed specialized KD methods like DKD and TTM.
- Base-to-New Generalization: In "zero-shot" style settings where a model is trained on base classes and tested on new ones, ABKD showed significantly better generalization than standard CLIP-based fine-tuning.
- Scaling: The benefits of ABKD actually increased as the teacher model got larger, suggesting it effectively bridges the "capacity gap" between massive teachers and tiny students.
Limitations and Future Work
- Hyperparameter Sensitivity: While the authors provide clear guidelines (e.g., use larger $\beta$ for high-dimensional outputs like text), $\alpha$ and $\beta$ still need to be tuned for optimal performance on new architectures.
- Static vs. Dynamic: Currently, $\alpha$ and $\beta$ are fixed throughout training. The authors suggest that future work could explore dynamic schedules—perhaps starting with a mode-covering $\alpha$ to explore the distribution and shifting to a mode-seeking $\alpha$ to sharpen the final model.
Summary updated