Multi-Teacher Knowledge Distillation for Incremental Implicitly-Refined Classification
TL;DR
Standard incremental learning often forgets "the big picture" when learning specific details. This paper introduces Multi-Teacher Knowledge Distillation (MTKD) to solve this in hierarchical settings, using the very first model as a "superclass expert" and the previous model as a "general expert." Combined with a Top-k prediction filter, it significantly boosts accuracy on complex, evolving datasets.
The Challenge of Hierarchical Forgetting
In standard incremental learning, a model learns new, distinct classes over time (e.g., first "Dog," then "Car"). However, real-world data is often hierarchical. The authors tackle Incremental Implicitly-Refined Classification (IIRC), where a model might first learn a broad superclass (e.g., "Bear") and later learn specific subclasses (e.g., "Polar Bear," "Grizzly Bear").
The core challenge is a specific form of "catastrophic forgetting." When the model learns a specific subclass, the new data is so similar to the old superclass data that the model effectively "overwrites" its broad knowledge. It loses the ability to recognize the general category "Bear" because it becomes hyper-focused on the "Polar Bear" label. Existing methods that only distill knowledge from the immediately preceding model fail because that model may have already begun to forget the superclass.
Methodology: Multi-Teacher Knowledge Distillation (MTKD)
To combat this, the authors propose Multi-Teacher Knowledge Distillation (MTKD). Instead of relying on a single teacher, the student model learns from two distinct experts:
- The Superclass Teacher ($M_0$): This is the initial model trained at step 0. Since it was trained before any subclasses were introduced, it possesses the "purest" knowledge of broad categories. It ensures the student retains the ability to distinguish superclasses.
- The General Teacher ($M_{t-1}$): This is the model from the immediately preceding step. It provides knowledge about all classes learned up to that point.
Additionally, the authors introduce a Top-k Prediction Restriction. Distilling from multiple teachers can introduce redundancy, causing the model to predict too many labels for a single image. Since IIRC implies a hierarchy (e.g., an image is both a "Bear" and a "Polar Bear"), the model is restricted to its top-$k$ highest-scoring predictions (typically $k=2$) to prune "noise" and maintain clarity.
Key Insights
- The Initial Model is a Goldmine: While standard practice uses the latest model as a teacher, this research demonstrates that for hierarchical tasks, the earliest model is equally vital for preserving broad category knowledge.
- Preventing Boundary "Squeezing": Learning a subclass often "squeezes" the decision boundary of its parent superclass. MTKD acts as a "spacer," keeping these boundaries distinct and preventing the superclass knowledge from being overwritten.
- Efficiency: Storing one extra teacher model (the initial one) is shown to be more effective than using the same memory to store additional raw images (exemplars).
Results
The MTKD strategy serves as a "plug-and-play" enhancement for existing algorithms like iCaRL and PODNet. Evaluations on IIRC-CIFAR100 and IIRC-ImageNet120 showed:
- Significant Accuracy Gains: Performance improved by 11.5% on IIRC-CIFAR100 and a staggering 25.7% on IIRC-ImageNet120.
- Superclass Preservation: Unlike baseline models that stop predicting superclasses once subclasses appear, MTKD models maintain high accuracy across both levels of the hierarchy.
Conclusion and Future Work
MTKD effectively addresses the unique challenges of hierarchical incremental learning. While it requires some extra storage for the initial model during training, the performance gains are substantial. Future work may explore teacher ensembles or dynamic $k$-values for more complex, open-world hierarchies.
Summary updated