Confidence-Aware Multi-Teacher Knowledge Distillation

Hailin Zhang, Defang Chen, Can Wang

TL;DR

CA-MKD is a novel framework that improves student model performance by adaptively weighting contributions from multiple teacher models based on their "confidence"—measured by how closely their predictions match ground-truth labels. By filtering out low-quality or biased teacher advice at both the prediction and intermediate feature levels, it consistently outperforms existing multi-teacher distillation methods across various neural architectures.

What Problem Does This Paper Address?

Knowledge Distillation (KD) is a popular technique where a small "student" model learns to mimic a large, high-performing "teacher" model. To further boost performance, researchers often use Multi-Teacher KD (MKD), following the "wisdom of the crowd" principle.

However, existing MKD methods have a major flaw: they typically treat all teachers equally (averaging) or use "label-free" heuristics (like entropy) to weight them. If several teachers provide poor or biased predictions for a specific image, the student is "misled" by the majority. The authors argue that we shouldn't trust a teacher blindly; instead, we should use the ground-truth labels already available during training to identify which teacher is actually reliable for a given sample.

Methods

The authors propose Confidence-Aware Multi-teacher Knowledge Distillation (CA-MKD), which introduces two main components:

  1. Sample-wise Prediction Weighting: For every training sample, the algorithm calculates the Cross-Entropy loss between each teacher’s prediction and the ground-truth label. Teachers with lower loss (higher confidence) are assigned higher weights. This ensures the student focuses on the most accurate "soft targets."
  2. Intermediate Feature Weighting: Inspired by FitNets, the authors extend this logic to internal layers. They map student features into the teacher's feature space and use the teacher's own classifier to see how well those features represent the correct class. This creates a "similarity measure" that helps the student decide which teacher's internal representations are most worth imitating.

The final loss function is a weighted combination of standard Cross-Entropy (with ground truth), the weighted KD loss (predictions), and the weighted intermediate feature loss.

Datasets

The researchers evaluated CA-MKD on three standard image classification benchmarks:

  • CIFAR-100: A dataset of 60,000 images across 100 classes.
  • Dogs: A fine-grained dataset containing high-resolution images of various dog breeds.
  • TinyImageNet: A subset of ImageNet with 200 classes at a $64 \times 64$ resolution.

They tested various architecture combinations, including ResNet, VGG, ShuffleNet, and MobileNet.

Key Insights and Ideas

  • Ground Truth as a Filter: While KD is often about learning "dark knowledge" (the relationships between incorrect classes), the ground-truth label remains the ultimate anchor. Using it to score teacher reliability prevents the student from learning "incorrect knowledge" from biased teachers.
  • Avoiding the Entropy Trap: Previous methods often favored teachers with "sharp" (low entropy) distributions. However, a teacher can be confidently wrong. CA-MKD avoids this by prioritizing accuracy over certainty.
  • Feature Space Alignment: The paper highlights that a teacher's internal features are only useful if they help the student eventually classify the image correctly. By passing student features through the teacher's classifier, they create a functional metric for feature-level distillation.

Key Results

  • State-of-the-Art Performance: CA-MKD consistently outperformed all compared MKD methods (like EBKD and AEKD). On CIFAR-100, it achieved an average improvement of 0.81% over the second-best method, with gains as high as 1.66% in specific teacher-student pairings.
  • Versatility: The method works even when teachers have different architectures (e.g., using a mix of ResNet8x4, ResNet20x4, and ResNet32x4 to teach a VGG8 student).
  • Ablation Success: Removing either the intermediate layer distillation or the adaptive weighting scheme led to a significant drop in accuracy, proving that both the "what to learn" (features) and "who to trust" (weighting) components are vital.

Limitations and Future Work

  • Computational Overhead: Training against multiple teachers is inherently more resource-intensive than single-teacher KD. While the authors only use the features from the last block to save costs, the total memory footprint during training remains high.
  • Dependence on Labels: Because the weighting mechanism relies on ground-truth labels, this specific formulation cannot be directly applied to unsupervised or self-supervised distillation tasks without modification.
  • Hyperparameter Sensitivity: The model introduces two new hyperparameters ($\alpha$ and $\beta$) to balance the losses. While the authors found stable values for their experiments, these might require tuning for significantly different datasets or tasks.

Summary updated

Abstract

Knowledge distillation is initially introduced to utilize additional supervision from a single teacher model for the student model training. To boost the student performance, some recent variants attempt to exploit diverse knowledge sources from multiple teachers. However, existing studies mainly integrate knowledge from diverse sources by averaging over multiple teacher predictions or combining them using other label-free strategies, which may mislead student in the presence of low-quality teacher predictions. To tackle this problem, we propose Confidence-Aware Multi-teacher Knowledge Distillation (CA-MKD), which adaptively assigns sample-wise reliability for each teacher prediction with the help of ground-truth labels, with those teacher predictions close to one-hot labels assigned large weights. Besides, CA-MKD incorporates features in intermediate layers to stable the knowledge transfer process. Extensive experiments show our CA-MKD consistently outperforms all compared state-of-the-art methods across various teacher-student architectures. Code is available: https://github.com/Rorozhl/CA-MKD.
BibTeX
@inproceedings{zhang2022confidence,
  author = {Hailin Zhang and Defang Chen and Can Wang},
  title = {{Confidence-Aware Multi-Teacher Knowledge Distillation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9747534},
}

Loading related content...

Figure