Abstract
Knowledge distillation is initially introduced to utilize additional supervision from a single teacher model for the student model training. To boost the student performance, some recent variants attempt to exploit diverse knowledge sources from multiple teachers. However, existing studies mainly integrate knowledge from diverse sources by averaging over multiple teacher predictions or combining them using other various label-free strategies, which may mislead student in the presence of low-quality teacher predictions. To tackle this problem, we propose Confidence-Aware Multi-teacher Knowledge Distillation (CA-MKD), which adaptively assigns sample-wise reliability for each teacher prediction with the help of ground-truth labels, with those teacher predictions close to one-hot labels assigned large weights. Besides, CA-MKD incorporates intermediate layers to stable the knowledge transfer process. Extensive experiments show that our CA-MKD consistently outperforms all compared state-of-the-art methods across various teacher-student architectures.
Introduction
Knowledge Distillation (KD) is a widely adopted technique for model compression, where a compact "student" model learns from a larger, high-performing "teacher" model. To further enhance the student's capabilities, researchers have turned to Multi-Teacher Knowledge Distillation (MKD), aiming to leverage the collective expertise of multiple models. However, a significant challenge in MKD is the risk of the student being misled by inaccurate or biased teacher predictions. Traditional MKD approaches often treat all teachers equally through simple averaging or use label-free heuristics like entropy to determine weights. These methods ignore the fact that a teacher's reliability can vary significantly depending on the specific input sample.
Methodology: The CA-MKD Framework
The proposed Confidence-Aware Multi-teacher Knowledge Distillation (CA-MKD) framework introduces a dynamic weighting mechanism that utilizes ground-truth labels to assess teacher reliability during training. The framework consists of two primary components:
- Sample-wise Prediction Weighting: For each training sample, CA-MKD calculates the Cross-Entropy loss between each teacher's prediction and the ground-truth label. Teachers whose predictions are closer to the one-hot ground-truth label (lower loss) are assigned higher weights. This ensures the student prioritizes high-quality "soft targets" and ignores potentially misleading advice.
- Intermediate Feature Weighting: To stabilize the distillation process, CA-MKD extends confidence-aware weighting to intermediate layers. Student features are mapped into the teacher's feature space and then passed through the teacher's own classifier. The resulting performance on the ground-truth label serves as a metric for how useful that teacher's internal representations are for the current sample.
The final training objective is a composite loss function combining standard Cross-Entropy with ground truth, weighted KD loss from predictions, and weighted intermediate feature loss.
Experimental Evaluation
The effectiveness of CA-MKD was validated across three standard benchmarks: CIFAR-100, the Dogs dataset, and TinyImageNet. The researchers tested the framework using various architecture pairings, including ResNet, VGG, ShuffleNet, and MobileNet.
Key Results
- Superior Performance: CA-MKD consistently outperformed state-of-the-art MKD methods such as EBKD and AEKD. On CIFAR-100, CA-MKD achieved an average improvement of 0.81% over the next best method.
- Scalability and Robustness: As shown in the performance plots, CA-MKD maintains its lead as the number of teachers increases, demonstrating better scalability than averaging-based methods which can plateau or degrade when poor teachers are added to the ensemble.
Conclusion
CA-MKD addresses a fundamental flaw in multi-teacher distillation by using ground-truth labels as an anchor for reliability. By adaptively weighting teacher contributions at both the prediction and feature levels, the framework ensures that the student model learns from the most accurate sources for every individual sample. While the method introduces some computational overhead due to the multi-teacher setup and requires ground-truth labels, its consistent performance gains across diverse architectures make it a powerful tool for high-performance model distillation.
Summary updated