Generalized Knowledge Distillation via Relationship Matching

Han-Jia Ye, Su Lu, De-Chuan Zhan

TL;DR

REFILLED decouples knowledge distillation into embedding and classifier stages, using relationship matching to bridge the gap between different label spaces. This "Generalized Knowledge Distillation" allows a student to learn from any well-trained teacher, even if their expertise lies in completely different categories.

What Problem Does This Paper Address?

Standard Knowledge Distillation (KD) is a powerful way to train efficient "student" models by mimicking a high-capacity "teacher." However, traditional KD has a major catch: the teacher and student must share the exact same set of classes. If you have a great model trained to recognize dog breeds, it’s not straightforward to use it to help a student learn to recognize birds.

This paper introduces Generalized Knowledge Distillation (GKD), a more flexible framework where the teacher's and student's label spaces can be identical, partially overlapping, or entirely disjoint. The core challenge is transferring "dark knowledge" without a shared vocabulary of labels.

An illustration of Standard, Cross-Task, and Generalized Knowledge Distillation.

Methods

The authors propose REFILLED (RElationship FacIlitated Local cLassifiEr Distillation), which breaks the distillation process into two distinct stages:

  1. Distill the Embedding via Comparison Matching: Instead of matching class probabilities, REFILLED matches how models compare instances. For a given "anchor" image, the student learns to mimic the teacher's relative similarity scores across a "tuple" of other images (positives and negatives). This transfers the teacher's underlying discriminative power without needing shared labels.
  2. Distill the Classifier Adaptively: To guide the student's final classification layer, the authors construct a temporary "embedding-induced classifier" based on the teacher's feature space. They introduce Local Knowledge Distillation (LKD), which focuses on the most relevant classes in a mini-batch, and an adaptive weighting mechanism. This weight ($\lambda_i$) automatically identifies which training samples the teacher is "confident" about, ensuring the student only receives high-quality, relevant guidance.
The REFILLED framework: Stage 1 (left) aligns similarity comparisons for embeddings; Stage 2 (right) aligns local classifier predictions with adaptive weights.

Datasets

The approach was evaluated on several standard computer vision benchmarks:

  • CUB-200-2011: A fine-grained bird classification dataset.
  • CIFAR-100: A general object recognition dataset.
  • Stanford Dogs: Used for "distant task" evaluation (distilling knowledge from birds to dogs).
  • MiniImageNet: A standard benchmark for few-shot and middle-shot learning.

Key Insights and Ideas

  • Relationship over Labels: The "aha moment" is that the ability to compare is a universal currency for knowledge. By matching similarity rankings (tuples) rather than absolute class scores, knowledge becomes portable across tasks.
  • Adaptive Supervision: Not all teacher advice is equal. The adaptive weight $\lambda_i$ acts as a "confidence filter," allowing the student to ignore the teacher when it's clearly out of its depth (e.g., a bird-expert teacher trying to label a car).
  • Local is Better: LKD shows that matching partial posteriors (only classes present in the current batch) is more effective than matching the full distribution when the number of classes is large, as it emphasizes the differences between the most relevant categories.

Key Results

  • Superiority in GKD: On CUB and CIFAR-100, REFILLED consistently outperformed vanilla KD and other relationship-based methods (like RKD) across all overlap ratios (0% to 100%).
  • Cross-Architecture Success: The method facilitates effective distillation even across different model families, such as transferring knowledge from a ResNet teacher to a MobileNet student.
  • Downstream Versatility: REFILLED achieved state-of-the-art results in one-step incremental learning and few-shot learning. For instance, in 5-way 30-shot learning on MiniImageNet, it reached 80.75% accuracy, significantly beating established meta-learning baselines like FEAT (78.84%).

Limitations and Future Work

  • Computational Overhead: Constructing and processing "semi-hard" tuples for comparison matching adds complexity and computational cost to the training loop compared to standard KD.
  • Teacher Quality: While robust, the student's performance is still fundamentally bounded by the teacher's discriminative ability in its own feature space.
  • Future Directions: The authors suggest exploring more complex relationship structures beyond simple tuples and extending the GKD framework to other modalities like text or audio.

Summary updated

Abstract

The knowledge of a well-trained deep neural network (a.k.a. the “teacher”) is valuable for learning similar tasks. Knowledge distillation extracts knowledge from the teacher and integrates it with the target model (a.k.a. the “student”), which expands the student’s knowledge and improves its learning efficacy. Instead of enforcing the teacher to work on the same task as the student, we borrow the knowledge from a teacher trained from a general label space — in this “Generalized Knowledge Distillation (GKD),” the classes of the teacher and the student may be the same, completely different, or partially overlapped . We claim that the comparison ability between instances acts as an essential factor threading knowledge across tasks, and propose the RE lationship F ac I litated L ocal c L assifi E r D istillation ( ReFilled ) approach, which decouples the GKD flow of the embedding and the top-layer classifier. In particular, different from reconciling the instance-label confidence between models, ReFilled requires the teacher to reweight the hard tuples pushed forward by the student and then matches the similarity comparison levels between instances. An embedding-induced classifier based on the teacher model supervises the student’s classification confidence and adaptively emphasizes the most related supervision from the teacher. ReFilled demonstrates strong discriminative ability when the classes of the teacher vary from the same to a fully non-overlapped set w.r.t. the student. It also achieves state-of-the-art performance on standard knowledge distillation, one-step incremental learning, and few-shot learning tasks.
BibTeX
@article{ye2023generalized,
  author = {Han-Jia Ye and Su Lu and De-Chuan Zhan},
  title = {{Generalized Knowledge Distillation via Relationship Matching}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.1109/tpami.2022.3160328},
}

Loading related content...

Figure