Semi-Online Knowledge Distillation

Zhiqiang Liu, Yanxia Liu, Chengkai HuangBMVC 2021 · OpenReview

TL;DR

This paper introduces Semi-Online Knowledge Distillation (SOKD), a novel framework that unifies Knowledge Distillation (KD) and Deep Mutual Learning (DML) to improve both student and teacher model performance. SOKD leverages the stable supervision signals from a well-trained teacher (KD) while enabling collaborative learning through peer-teaching (DML), leading to state-of-the-art results on CIFAR-100 and ImageNet.

What Problem Does This Paper Address?

Deep learning models, especially for computer vision tasks, have become increasingly powerful but also computationally expensive and memory-intensive. This makes deploying them on resource-constrained devices challenging. Knowledge Distillation (KD) is a popular model compression technique where a smaller "student" network learns from a larger "teacher" network. However, traditional KD is a one-way process where the student struggles to fully mimic the teacher due to a significant performance gap. Deep Mutual Learning (DML) offers an alternative by having multiple networks learn collaboratively, but it can suffer from unstable training signals in the early stages. This paper addresses the limitations of both KD and DML by proposing a unified framework that combines their strengths, aiming to improve both student performance and even enhance the teacher model.

Methods

The core contribution is the Semi-Online Knowledge Distillation (SOKD) method. SOKD builds upon the observation that KD provides stable teacher supervision, while DML's peer-teaching fashion helps students imitate more effectively.

The SOKD framework consists of three main components:

  1. Well-trained Teacher: A pre-trained teacher model.
  2. Knowledge Bridge Module (KBM): This module acts as an intermediary, taking the outputs of the teacher's low-level layers as input and aiming to capture the teacher's knowledge. The KBM's structure is inspired by CNN architectures, often mirroring the teacher's high-level layers.
  3. Student Network: The smaller network being trained.

During the training phase, the teacher model is fixed. The KBM and the student are updated simultaneously. The KBM learns to mimic the teacher's knowledge, and the student learns from the KBM. This peer-teaching approach alleviates the student's imitation difficulty. The training objectives involve minimizing KL divergence between the KBM's and student's outputs, as well as between the KBM's outputs and the teacher's outputs.

During the inference phase, the original teacher network is reconstructed by replacing its high-level layers with the trained KBM. This process surprisingly results in a more powerful teacher model. The paper also demonstrates that SOKD can be extended to feature-based distillation methods by adjusting the KBM to process intermediate feature maps instead of logits.

Datasets

The proposed SOKD method was evaluated on the following datasets:

  • CIFAR-100: Used for extensive experiments comparing SOKD with various offline and online knowledge distillation methods.
  • ImageNet: Used to demonstrate the generalization capabilities of SOKD on a large-scale dataset.

Key Insights and Ideas

  • KD and DML are Complementary: The paper's central insight is that traditional KD and DML, despite their individual shortcomings, are complementary. KD offers stable supervision, while DML provides a more effective imitation mechanism.
  • The Power of a Knowledge Bridge: The KBM acts as a crucial component, bridging the gap between the teacher's knowledge and the student's learning capacity. It allows for more nuanced knowledge transfer than direct imitation.
  • Teacher Improvement is a Bonus: A surprising finding is that the SOKD framework not only improves the student's performance but also enhances the teacher's accuracy by reconstructing it with the KBM. This suggests that the KBM can capture and refine valuable knowledge from the teacher.
  • Logit-Based Distillation Still Matters: Despite the rise of feature-based methods, the paper shows that logit-based distillation, when effectively implemented (as in SOKD), can still achieve superior performance, indicating that the potential of logit knowledge is often underestimated.

Key Results

  • CIFAR-100 Performance: SOKD consistently outperforms state-of-the-art offline and online knowledge distillation methods on CIFAR-100, achieving higher student accuracy.
  • Teacher Performance Improvement: SOKD improves the teacher's Top-1 accuracy by up to 1.87% on CIFAR-100 and 0.84% on ImageNet compared to the vanilla pre-trained teacher.
  • ImageNet Generalization: The method demonstrates strong performance on the large-scale ImageNet dataset, outperforming other methods.
  • Imitation Analysis: Experiments using CKA similarity, imitation error rate (IER), and misleading rate (MR) show that SOKD achieves higher imitation ability (higher CKA, lower IER) with less misleading information (lower MR) compared to DML.

Limitations and Future Work

  • KBM Design: While the paper explores different KBM structures, it notes that the KBM's performance is sensitive to its design. The optimal KBM structure might vary depending on the specific teacher-student architectures.
  • Computational Cost: While SOKD aims for efficiency, the introduction of the KBM adds some computational overhead.
  • Future Work: The authors suggest further exploring the KBM design and its impact on performance. They also highlight the potential for extending SOKD to other domains and tasks beyond image classification. The paper also hints at further optimizing the KBM for reduced computational cost while maintaining performance.

Summary updated

Abstract

Knowledge distillation is an effective and stable method for model compression via knowledge transfer. Conventional knowledge distillation (KD) is to transfer knowledge from a large and well pre-trained teacher network to a small student network, which is a one-way process. Recently, deep mutual learning (DML) has been proposed to help student networks learn collaboratively and simultaneously. However, to the best of our knowledge, KD and DML have never been jointly explored in a unified framework to solve the knowledge distillation problem. In this paper, we investigate that the teacher model supports more trustworthy supervision signals in KD, while the student captures more similar behaviors from the teacher in DML. Based on these observations, we first propose to combine KD with DML in a unified framework. Furthermore, we propose a Semi-Online Knowledge Distillation (SOKD) method that effectively improves the performance of the student and the teacher. In this method, we introduce the peer-teaching training fashion in DML in order to alleviate the student's imitation difficulty, and also leverage the supervision signals provided by the well-trained teacher in KD. Besides, we also show our framework can be easily extended to feature-based distillation methods. Extensive experiments on CIFAR-100 and ImageNet datasets demonstrate the proposed method achieves state-of-the-art performance.
BibTeX
@inproceedings{liu2021semi,
  author = {Zhiqiang Liu and Yanxia Liu and Chengkai Huang},
  title = {{Semi-Online Knowledge Distillation}},
  booktitle = {BMVC},
  year = {2021},
  url = {https://openreview.net/forum?id=Z14n2acl4A},
}
Figure