Abstract
The knowledge of a well-trained deep neural network (the "teacher") is valuable for learning similar tasks. Knowledge distillation (KD) extracts this knowledge and integrates it with a target model (the "student"), improving learning efficacy. Traditional KD requires the teacher and student to share the same label space. This paper introduces Generalized Knowledge Distillation (GKD), where the classes of the teacher and student may be identical, partially overlapping, or completely disjoint. We propose the RElationship FacIlitated Local cLassifiEr Distillation (REFILLED) approach, which decouples the GKD flow into embedding and top-layer classifier stages. REFILLED matches similarity comparison levels between instances and uses an embedding-induced classifier to supervise student confidence. REFILLED demonstrates strong discriminative ability across various label space overlaps and achieves state-of-the-art performance on standard KD, incremental learning, and few-shot learning tasks.
Introduction: From Standard to Generalized KD
Knowledge Distillation is a cornerstone technique for model compression and transfer learning. By training a compact student model to mimic the outputs of a larger, pre-trained teacher, researchers can produce efficient models that retain high performance. However, a significant constraint of standard KD is the requirement for a shared label space. If a teacher is an expert in dog breeds, its "dark knowledge" is not easily transferred to a student learning to classify birds using traditional methods.
Generalized Knowledge Distillation (GKD) addresses this by allowing the teacher and student to operate on different tasks. The core hypothesis is that the ability to compare instances—determining which images are more similar than others—is a fundamental form of knowledge that transcends specific category labels.
The REFILLED Approach
To implement GKD, the authors propose REFILLED, a two-stage framework that separates the distillation of feature embeddings from the distillation of the classification layer.
1. Distilling the Embedding via Relationship Matching
Instead of matching class probabilities, REFILLED focuses on the structural relationships within the feature space. The student is tasked with mimicking the teacher's relative similarity rankings. For any given "anchor" image, the student learns to replicate how the teacher compares it to a "tuple" of other images. By matching these similarity comparison levels, the student inherits the teacher's underlying discriminative power without needing to understand the teacher's specific labels.
2. Distilling the Classifier Adaptively
To guide the student's final classification layer, REFILLED constructs a temporary "embedding-induced classifier" based on the teacher's features. This stage utilizes Local Knowledge Distillation (LKD), which focuses on the most relevant classes within a mini-batch rather than the entire label space.
Crucially, an adaptive weighting mechanism ($\lambda_i$) is introduced. This weight identifies training samples where the teacher is highly confident and relevant to the student's task. This ensures the student ignores "noisy" or irrelevant guidance, such as when a bird-expert teacher provides low-confidence predictions for a non-bird image.
Experimental Results
The REFILLED method was rigorously tested on benchmarks including CUB-200-2011, CIFAR-100, Stanford Dogs, and MiniImageNet.
Performance in GKD
On the CUB and CIFAR-100 datasets, REFILLED consistently outperformed both vanilla KD and existing relationship-based methods (like RKD). Its superiority held true across the entire spectrum of label overlap—from 0% (completely disjoint tasks) to 100% (standard KD). This demonstrates that relationship matching is a robust bridge for cross-task knowledge transfer.
Versatility and Downstream Tasks
Beyond standard classification, REFILLED showed remarkable success in:
- Cross-Architecture Distillation: Successfully transferring knowledge from high-capacity ResNet teachers to lightweight MobileNet students.
- Few-Shot Learning: On the MiniImageNet 5-way 30-shot task, REFILLED achieved 80.75% accuracy, significantly surpassing established meta-learning baselines like FEAT (78.84%).
- Incremental Learning: The method proved effective in one-step incremental learning scenarios, maintaining performance as new classes are introduced.
Conclusion
REFILLED redefines knowledge distillation by shifting the focus from label-matching to relationship-matching. By decoupling the embedding and classifier stages and employing adaptive supervision, it enables students to learn from any well-trained teacher, regardless of their specific expertise. While the construction of "semi-hard" tuples for comparison adds some computational overhead, the resulting gains in flexibility and accuracy make REFILLED a powerful tool for modern deep learning workflows.
Summary updated