SMKD: Selective Mutual Knowledge Distillation

Ziyun Li, Xinshao Wang, Neil M. Robertson, David A. Clifton, Christoph Meinel, Haojin YangIJCNN 2023 · OpenReview

TL;DR

This paper introduces Selective Mutual Knowledge Distillation (SMKD), a novel framework that addresses the problem of unreliable knowledge transfer in collaborative model training. By selectively distilling only confident knowledge, SMKD significantly improves robustness, especially in the presence of noisy labels, outperforming existing methods.

What Problem Does This Paper Address?

In collaborative learning scenarios, multiple models train together, sharing their "knowledge" to improve overall performance. This technique, known as Mutual Knowledge Distillation (MKD), assumes that all knowledge shared is beneficial. However, this assumption breaks down when dealing with noisy labels in datasets. Noisy labels can lead to models learning incorrect information, which then gets propagated to other models during distillation, hindering rather than helping the learning process. While methods exist to make individual models more robust to noise, the problem of selectively distilling only the reliable knowledge from these models has been largely overlooked. This paper tackles this gap by proposing a framework that intelligently filters knowledge before it's shared.

Methods

The core innovation is the Selective Mutual Knowledge Distillation (SMKD) framework. SMKD introduces a generic knowledge selection formulation that determines which pieces of knowledge are "confident enough" to be distilled. This selection is primarily based on the entropy of a model's predictions: lower entropy indicates higher confidence.

SMKD offers two main modes for this selection:

  • SMKD-Static (SMKD-S): Uses a fixed confidence threshold throughout training. This mode can degenerate into conventional MKD (distilling all knowledge) or zero knowledge distillation (distilling no knowledge) under specific parameter settings.
  • SMKD-Progressive (SMKD-P): Dynamically adjusts the confidence threshold during training. This allows the system to become more selective as models gain confidence, or to relax selectivity as training progresses. The paper proposes a flexible formula for this threshold adjustment, controlled by parameters b₂ and η.

Crucially, SMKD is designed to be compatible with existing self-training and label modification techniques, allowing for a synergistic enhancement of robustness.

Datasets

The paper evaluates SMKD on several standard datasets:

  • CIFAR-100: A dataset with 100 classes, used with both synthetic and real-world label noise.
  • Food-101: A real-world dataset known for its noisy labels.
  • Webvision: Another real-world dataset with noisy labels, derived from ImageNet.

The authors also mention using a subset of ImageNet for Webvision, following a "Mini" setting from prior work.

Key Insights and Ideas

  • Not all knowledge is created equal: The fundamental insight is that in noisy environments, blindly distilling all knowledge is detrimental. Selective distillation of confident knowledge is key to robust learning.
  • Entropy as a confidence measure: Using prediction entropy is an effective and intuitive way to quantify a model's confidence in its predictions, serving as the basis for knowledge selection.
  • Progressive selectivity is beneficial: Dynamically adjusting the knowledge selection threshold during training (SMKD-P) generally yields better results than a static threshold, suggesting that models' confidence evolves and should be accounted for.
  • Synergy with label correction: SMKD can be effectively combined with methods that correct noisy labels (like ProSelfLC and MyLC), leading to even greater improvements in robustness. This highlights SMKD's role as a complementary technique.

Key Results

  • Significant improvement under label noise: SMKD consistently outperforms conventional MKD and other state-of-the-art label noise robust learning methods across various noise rates and datasets. For instance, on CIFAR-100 with 40% symmetric noise, SMKD-P combined with ProSelfLC or MyLC achieved around 5-6% higher accuracy than other leading methods.
  • SMKD-P outperforms SMKD-S: The progressive version of SMKD generally shows better performance, indicating the advantage of adaptive knowledge selection.
  • Effective across different architectures: SMKD demonstrates its effectiveness when training models with the same or different architectures.
  • Hyperparameter analysis confirms design: Experiments show that decreasing the confidence threshold (larger η) generally leads to better accuracy, and the progressive adjustment of the threshold (controlled by b₂) is crucial for optimal performance.

Limitations and Future Work

  • Hyperparameter sensitivity: While the paper explores hyperparameters like b₂ and η, finding the optimal settings might still require careful tuning for different tasks and datasets.
  • Computational overhead: While SMKD aims to improve distillation, the added selection mechanism might introduce a slight computational overhead compared to standard MKD, though the paper doesn't explicitly quantify this.
  • Future work: The authors suggest exploring more sophisticated knowledge selection criteria beyond simple entropy, and investigating SMKD in other domains beyond image classification. They also highlight the potential for SMKD in fine-tuning and incremental learning scenarios.

Summary updated

Abstract

Mutual knowledge distillation (MKD) is a technique used to transfer knowledge between multiple models in a collaborative manner. However, it is important to note that not all knowledge is accurate or reliable, particularly under challenging conditions such as label noise, which can lead to models that memorize undesired information. This problem can be addressed by improving the reliability of the knowledge source, as well as selectively selecting reliable knowledge for distillation. While making a model more reliable is a widely studied topic, selective MKD has received less attention. To address this, we propose a new framework called selective mutual knowledge distillation (SMKD). The key component of SMKD is a generic knowledge selection formulation, which allows for either static or progressive selection thresholds. Additionally, SMKD covers two special cases: using no knowledge and using all knowledge, resulting in a unified MKD framework. We present extensive experimental results to demonstrate the effectiveness of SMKD and justify its design.
BibTeX
@inproceedings{li2023smkd,
  author = {Ziyun Li and Xinshao Wang and Neil M. Robertson and David A. Clifton and Christoph Meinel and Haojin Yang},
  title = {{SMKD: Selective Mutual Knowledge Distillation}},
  booktitle = {IJCNN},
  year = {2023},
  url = {https://openreview.net/forum?id=NQI4AJA1GT},
}
Figure