Self-distilled Knowledge Delegator for Exemplar-free Class Incremental Learning

Fanfan Ye, Liang Ma, Qiaoyong Zhong, Di Xie, Shiliang Pu

TL;DR

Researchers from Hikvision have developed a Self-distilled Knowledge Delegator (SKD) that solves the "catastrophic forgetting" problem in AI without needing to store any old data. By using a generative model to "delegate" knowledge from old versions of a network to new ones through synthetic samples, they achieve state-of-the-art performance in exemplar-free incremental learning, even rivaling methods that actually keep old data.

What Problem Does This Paper Address?

Deep Neural Networks are notoriously bad at multitasking over time. When you train a model on Task A and then move to Task B, it usually "forgets" Task A—a phenomenon called catastrophic forgetting.

The standard fix is to keep a small "memory" of old data (exemplars) to replay during new training. However, in the real world, privacy laws (like GDPR) or data security often forbid storing old user data. This leads to Exemplar-free Class Incremental Learning (CIL). Existing exemplar-free methods usually rely on "regularization" (freezing important weights), but these tend to fail as the number of tasks grows. The challenge is: how do we preserve the "essence" of what a model learned without actually keeping the data it learned from?

Methods

The authors introduce the Self-distilled Knowledge Delegator (SKD). Instead of saving data, they train a generative network to act as a "messenger" of knowledge. The process happens in two stages for every new task:

  1. Stage One: Training the Delegator (Imitate & Explore):
    • The goal is to train the SKD (a generator) to produce synthetic data that captures the feature space of the old model ($T_n$).
  2. Imitate Phase: A "naive" clone of the model ($S_n$) tries to match the old model's outputs using the generated data.
  3. Explore Phase: This is the "adversarial" part. The SKD is pushed to generate data that makes the two models disagree. This forces the generator to find the "hard" boundaries and diverse features of the old classes, rather than just generating the same easy sample over and over.
  4. Helper BN Layer: They add a specific Batch Normalization layer to the generator to help bridge the "domain gap" between synthetic noise and real-world data distributions.
  5. Stage Two: Incremental Learning:
    • The system combines the synthetic data from the SKD (representing old tasks) with real data from the new task.
  6. They use a Feature Cosine Discrepancy loss to ensure the new model's internal representations of old classes don't drift.

Datasets

The method was evaluated on four widely recognized computer vision benchmarks:

  • CIFAR-100: 100 classes of low-resolution (32x32) images.
  • ImageNet-Subset: 100 classes sampled from the massive ImageNet dataset (224x224).
  • Caltech-101: A classic object recognition dataset.
  • Flowers-102: A fine-grained classification dataset (many similar-looking flower species).

Key Insights and Ideas

  • Feature Space over Image Quality: Unlike typical GANs that try to make "pretty" pictures, the SKD doesn't care if the generated images look like garbage to a human. It only cares that the images trigger the same internal features in the neural network as the original data.
  • Adversarial Exploration is Key: Simply distilling knowledge isn't enough. By forcing the generator to "explore" (create samples that the current model hasn't mastered yet), the SKD covers the entire distribution of the old data more effectively.
  • The "Helper BN" Trick: One of the biggest hurdles in data-free learning is that synthetic data has different statistics than real data. The "Helper BN" layer acts as a translator, normalizing the generator's output so the main model can process it alongside real new data without getting confused.

Key Results

  • Superiority in its Class: On CIFAR-100 (10-task setting), SKD achieved 59.62% average accuracy, outperforming the previous best exemplar-free method (SDC+EWC) by 3.53%.
  • Beating the "Memory" Methods: Remarkably, SKD outperformed several "exemplar-based" methods (like iCaRL and BiC) that actually have access to old data, despite SKD being completely data-free.
  • High-Resolution Success: While previous data-free methods usually only worked on tiny MNIST or CIFAR-10 images, SKD proved effective on high-resolution datasets like ImageNet-Subset and fine-grained tasks like Flowers-102.

Limitations and Future Work

The authors note that as the number of incremental tasks increases (e.g., moving from 5 tasks to 20 tasks), the performance still declines. This is likely because the "delegator" itself has to be trained based on a model that has already undergone several rounds of incremental learning, leading to a "compounding error" or "knowledge dilution" effect.

Future work will likely focus on making the SKD more robust to these long-term sequences and exploring how this "delegation" concept could apply to other fields like unsupervised domain adaptation.

Summary updated

Abstract

Exemplar-free incremental learning is extremely challenging due to inaccessibility of data from old tasks. In this paper, we attempt to exploit the knowledge encoded in a previously trained classification model to handle the catas-trophic forgetting problem in continual learning. Specifically, we introduce a so-called knowledge delegator, which is capable of transferring knowledge from the trained model to a randomly re-initialized new model by generating informative samples. Given the previous model only, the delegator is effectively learned using a self-distillation mechanism in a data-free manner. The knowledge extracted by the delegator is then utilized to maintain the performance of the model on old tasks in incremental learning. This simple incremental learning framework surpasses existing exemplar-free methods by a large margin on four widely used class incremental benchmarks, namely CIFAR-100, ImageNet-Subset, Caltech-101 and Flowers-102. Notably, we achieve comparable performance to some exemplar-based methods without accessing any exemplars.
BibTeX
@inproceedings{ye2022self,
  author = {Fanfan Ye and Liang Ma and Qiaoyong Zhong and Di Xie and Shiliang Pu},
  title = {{Self-distilled Knowledge Delegator for Exemplar-free Class Incremental Learning}},
  booktitle = {IJCNN},
  year = {2022},
  url = {https://openreview.net/forum?id=LZ3XEJ7Aqd},
}

Loading related content...

Figure