TL;DR
REFILLED decouples knowledge distillation into embedding and classifier stages, using relationship matching to bridge the gap between different label spaces. This "Generalized Knowledge Distillation" allows a student to learn from any well-trained teacher, even if their expertise lies in completely different categories.
What Problem Does This Paper Address?
Standard Knowledge Distillation (KD) is a powerful way to train efficient "student" models by mimicking a high-capacity "teacher." However, traditional KD has a major catch: the teacher and student must share the exact same set of classes. If you have a great model trained to recognize dog breeds, it’s not straightforward to use it to help a student learn to recognize birds.
This paper introduces Generalized Knowledge Distillation (GKD), a more flexible framework where the teacher's and student's label spaces can be identical, partially overlapping, or entirely disjoint. The core challenge is transferring "dark knowledge" without a shared vocabulary of labels.
Methods
The authors propose REFILLED (RElationship FacIlitated Local cLassifiEr Distillation), which breaks the distillation process into two distinct stages:
- Distill the Embedding via Comparison Matching: Instead of matching class probabilities, REFILLED matches how models compare instances. For a given "anchor" image, the student learns to mimic the teacher's relative similarity scores across a "tuple" of other images (positives and negatives). This transfers the teacher's underlying discriminative power without needing shared labels.
- Distill the Classifier Adaptively: To guide the student's final classification layer, the authors construct a temporary "embedding-induced classifier" based on the teacher's feature space. They introduce Local Knowledge Distillation (LKD), which focuses on the most relevant classes in a mini-batch, and an adaptive weighting mechanism. This weight ($\lambda_i$) automatically identifies which training samples the teacher is "confident" about, ensuring the student only receives high-quality, relevant guidance.
Datasets
The approach was evaluated on several standard computer vision benchmarks:
- CUB-200-2011: A fine-grained bird classification dataset.
- CIFAR-100: A general object recognition dataset.
- Stanford Dogs: Used for "distant task" evaluation (distilling knowledge from birds to dogs).
- MiniImageNet: A standard benchmark for few-shot and middle-shot learning.
Key Insights and Ideas
- Relationship over Labels: The "aha moment" is that the ability to compare is a universal currency for knowledge. By matching similarity rankings (tuples) rather than absolute class scores, knowledge becomes portable across tasks.
- Adaptive Supervision: Not all teacher advice is equal. The adaptive weight $\lambda_i$ acts as a "confidence filter," allowing the student to ignore the teacher when it's clearly out of its depth (e.g., a bird-expert teacher trying to label a car).
- Local is Better: LKD shows that matching partial posteriors (only classes present in the current batch) is more effective than matching the full distribution when the number of classes is large, as it emphasizes the differences between the most relevant categories.
Key Results
- Superiority in GKD: On CUB and CIFAR-100, REFILLED consistently outperformed vanilla KD and other relationship-based methods (like RKD) across all overlap ratios (0% to 100%).
- Cross-Architecture Success: The method facilitates effective distillation even across different model families, such as transferring knowledge from a ResNet teacher to a MobileNet student.
- Downstream Versatility: REFILLED achieved state-of-the-art results in one-step incremental learning and few-shot learning. For instance, in 5-way 30-shot learning on MiniImageNet, it reached 80.75% accuracy, significantly beating established meta-learning baselines like FEAT (78.84%).
Limitations and Future Work
- Computational Overhead: Constructing and processing "semi-hard" tuples for comparison matching adds complexity and computational cost to the training loop compared to standard KD.
- Teacher Quality: While robust, the student's performance is still fundamentally bounded by the teacher's discriminative ability in its own feature space.
- Future Directions: The authors suggest exploring more complex relationship structures beyond simple tuples and extending the GKD framework to other modalities like text or audio.
Summary updated