Multi-Teacher Knowledge Distillation for Incremental Implicitly-Refined Classification

Longhui Yu, Zhenyu Weng, Yuqing Wang, Yuesheng Zhu

Multi-Teacher Knowledge Distillation for Incremental Implicitly-Refined Classification

TL;DR

Standard incremental learning often forgets "the big picture" when learning specific details. This paper introduces Multi-Teacher Knowledge Distillation (MTKD) to solve this in hierarchical settings, using the very first model as a "superclass expert" and the previous model as a "general expert." Combined with a Top-k prediction filter, it significantly boosts accuracy on complex, evolving datasets.

The Challenge of Hierarchical Forgetting

In standard incremental learning, a model learns new, distinct classes over time (e.g., first "Dog," then "Car"). However, real-world data is often hierarchical. The authors tackle Incremental Implicitly-Refined Classification (IIRC), where a model might first learn a broad superclass (e.g., "Bear") and later learn specific subclasses (e.g., "Polar Bear," "Grizzly Bear").

Visualization of learning subclasses within a superclass region, illustrating the hierarchical nature of IIRC.

The core challenge is a specific form of "catastrophic forgetting." When the model learns a specific subclass, the new data is so similar to the old superclass data that the model effectively "overwrites" its broad knowledge. It loses the ability to recognize the general category "Bear" because it becomes hyper-focused on the "Polar Bear" label. Existing methods that only distill knowledge from the immediately preceding model fail because that model may have already begun to forget the superclass.

Methodology: Multi-Teacher Knowledge Distillation (MTKD)

To combat this, the authors propose Multi-Teacher Knowledge Distillation (MTKD). Instead of relying on a single teacher, the student model learns from two distinct experts:

  1. The Superclass Teacher ($M_0$): This is the initial model trained at step 0. Since it was trained before any subclasses were introduced, it possesses the "purest" knowledge of broad categories. It ensures the student retains the ability to distinguish superclasses.
  2. The General Teacher ($M_{t-1}$): This is the model from the immediately preceding step. It provides knowledge about all classes learned up to that point.
Comparison between standard single-teacher distillation (top) and the proposed Multi-Teacher Knowledge Distillation (bottom).

Additionally, the authors introduce a Top-k Prediction Restriction. Distilling from multiple teachers can introduce redundancy, causing the model to predict too many labels for a single image. Since IIRC implies a hierarchy (e.g., an image is both a "Bear" and a "Polar Bear"), the model is restricted to its top-$k$ highest-scoring predictions (typically $k=2$) to prune "noise" and maintain clarity.

Key Insights

  • The Initial Model is a Goldmine: While standard practice uses the latest model as a teacher, this research demonstrates that for hierarchical tasks, the earliest model is equally vital for preserving broad category knowledge.
  • Preventing Boundary "Squeezing": Learning a subclass often "squeezes" the decision boundary of its parent superclass. MTKD acts as a "spacer," keeping these boundaries distinct and preventing the superclass knowledge from being overwritten.
  • Efficiency: Storing one extra teacher model (the initial one) is shown to be more effective than using the same memory to store additional raw images (exemplars).

Results

The MTKD strategy serves as a "plug-and-play" enhancement for existing algorithms like iCaRL and PODNet. Evaluations on IIRC-CIFAR100 and IIRC-ImageNet120 showed:

  • Significant Accuracy Gains: Performance improved by 11.5% on IIRC-CIFAR100 and a staggering 25.7% on IIRC-ImageNet120.
  • Superclass Preservation: Unlike baseline models that stop predicting superclasses once subclasses appear, MTKD models maintain high accuracy across both levels of the hierarchy.

Conclusion and Future Work

MTKD effectively addresses the unique challenges of hierarchical incremental learning. While it requires some extra storage for the initial model during training, the performance gains are substantial. Future work may explore teacher ensembles or dynamic $k$-values for more complex, open-world hierarchies.

Summary updated

Abstract

Incremental learning methods can learn new classes continually by distilling knowledge from the last model (as a teacher model) to the current model (as a student model) in the sequentially learning process. However, these methods cannot work for Incremental Implicitly-Refined Classification (IIRC), an incremental learning extension where the incoming classes could have two granularity levels, a superclass label and a subclass label. This is because the previously learned superclass knowledge may be occupied by the sub-class knowledge learned sequentially. To solve this problem, we propose a novel Multi-Teacher Knowledge Distillation (MTKD) strategy. To preserve the subclass knowledge, we use the last model as a general teacher to distill the previous knowledge for the student model. To preserve the superclass knowledge, we use the initial model as a superclass teacher to distill the superclass knowledge as the initial model contains abundant superclass knowledge. However, distilling knowledge from two teacher models could result in the student model making some redundant predictions. We further propose a post-processing mechanism, called as Top-k prediction restriction to reduce the redundant predictions. Our experimental results on IIRC-ImageNet120 and IIRC-CIFAR100 show that the proposed method can achieve better classification accuracy compared with existing state-of-the-art methods.
BibTeX
@inproceedings{yu2022multi,
  author = {Longhui Yu and Zhenyu Weng and Yuqing Wang and Yuesheng Zhu},
  title = {{Multi-Teacher Knowledge Distillation for Incremental Implicitly-Refined Classification}},
  booktitle = {ICME},
  year = {2022},
  url = {https://openreview.net/forum?id=hm4O2aFzbgc},
}
Figure