TL;DR
This paper introduces Selective Mutual Knowledge Distillation (SMKD), a novel framework that addresses the problem of unreliable knowledge transfer in collaborative model training. By selectively distilling only confident knowledge, SMKD significantly improves robustness, especially in the presence of noisy labels, outperforming existing methods.
What Problem Does This Paper Address?
In collaborative learning scenarios, multiple models train together, sharing their "knowledge" to improve overall performance. This technique, known as Mutual Knowledge Distillation (MKD), assumes that all knowledge shared is beneficial. However, this assumption breaks down when dealing with noisy labels in datasets. Noisy labels can lead to models learning incorrect information, which then gets propagated to other models during distillation, hindering rather than helping the learning process. While methods exist to make individual models more robust to noise, the problem of selectively distilling only the reliable knowledge from these models has been largely overlooked. This paper tackles this gap by proposing a framework that intelligently filters knowledge before it's shared.
Methods
The core innovation is the Selective Mutual Knowledge Distillation (SMKD) framework. SMKD introduces a generic knowledge selection formulation that determines which pieces of knowledge are "confident enough" to be distilled. This selection is primarily based on the entropy of a model's predictions: lower entropy indicates higher confidence.
SMKD offers two main modes for this selection:
- SMKD-Static (SMKD-S): Uses a fixed confidence threshold throughout training. This mode can degenerate into conventional MKD (distilling all knowledge) or zero knowledge distillation (distilling no knowledge) under specific parameter settings.
- SMKD-Progressive (SMKD-P): Dynamically adjusts the confidence threshold during training. This allows the system to become more selective as models gain confidence, or to relax selectivity as training progresses. The paper proposes a flexible formula for this threshold adjustment, controlled by parameters
b₂andη.
Crucially, SMKD is designed to be compatible with existing self-training and label modification techniques, allowing for a synergistic enhancement of robustness.
Datasets
The paper evaluates SMKD on several standard datasets:
- CIFAR-100: A dataset with 100 classes, used with both synthetic and real-world label noise.
- Food-101: A real-world dataset known for its noisy labels.
- Webvision: Another real-world dataset with noisy labels, derived from ImageNet.
The authors also mention using a subset of ImageNet for Webvision, following a "Mini" setting from prior work.
Key Insights and Ideas
- Not all knowledge is created equal: The fundamental insight is that in noisy environments, blindly distilling all knowledge is detrimental. Selective distillation of confident knowledge is key to robust learning.
- Entropy as a confidence measure: Using prediction entropy is an effective and intuitive way to quantify a model's confidence in its predictions, serving as the basis for knowledge selection.
- Progressive selectivity is beneficial: Dynamically adjusting the knowledge selection threshold during training (SMKD-P) generally yields better results than a static threshold, suggesting that models' confidence evolves and should be accounted for.
- Synergy with label correction: SMKD can be effectively combined with methods that correct noisy labels (like ProSelfLC and MyLC), leading to even greater improvements in robustness. This highlights SMKD's role as a complementary technique.
Key Results
- Significant improvement under label noise: SMKD consistently outperforms conventional MKD and other state-of-the-art label noise robust learning methods across various noise rates and datasets. For instance, on CIFAR-100 with 40% symmetric noise, SMKD-P combined with ProSelfLC or MyLC achieved around 5-6% higher accuracy than other leading methods.
- SMKD-P outperforms SMKD-S: The progressive version of SMKD generally shows better performance, indicating the advantage of adaptive knowledge selection.
- Effective across different architectures: SMKD demonstrates its effectiveness when training models with the same or different architectures.
- Hyperparameter analysis confirms design: Experiments show that decreasing the confidence threshold (larger
η) generally leads to better accuracy, and the progressive adjustment of the threshold (controlled byb₂) is crucial for optimal performance.
Limitations and Future Work
- Hyperparameter sensitivity: While the paper explores hyperparameters like
b₂andη, finding the optimal settings might still require careful tuning for different tasks and datasets. - Computational overhead: While SMKD aims to improve distillation, the added selection mechanism might introduce a slight computational overhead compared to standard MKD, though the paper doesn't explicitly quantify this.
- Future work: The authors suggest exploring more sophisticated knowledge selection criteria beyond simple entropy, and investigating SMKD in other domains beyond image classification. They also highlight the potential for SMKD in fine-tuning and incremental learning scenarios.
Summary updated