Abstract
Exemplar-free incremental learning is extremely challenging due to the inaccessibility of data from old tasks. In this paper, we attempt to exploit the knowledge encoded in a previously trained classification model to handle the catastrophic forgetting problem in continual learning. Specifically, we introduce a so-called knowledge delegator, which is capable of transferring knowledge from the trained model to a randomly re-initialized new model by generating informative samples. Given the previous model only, the delegator is effectively learned using a self-distillation mechanism in a data-free manner. The knowledge extracted by the delegator is then utilized to maintain the performance of the model on old tasks in incremental learning. This simple incremental learning framework surpasses existing exemplar-free methods by a large margin on four widely used class incremental benchmarks, namely CIFAR-100, ImageNet-Subset, Caltech-101 and Flowers-102. Notably, we achieve comparable performance to some exemplar-based methods without accessing any exemplars.
Introduction
Deep Neural Networks (DNNs) typically suffer from catastrophic forgetting when trained on a sequence of tasks. While learning new information (Task B), the model overwrites the weights essential for previous knowledge (Task A). The standard solution in Class Incremental Learning (CIL) involves maintaining a small memory buffer of old data (exemplars) to replay during new training sessions. However, in many real-world scenarios, data privacy regulations (such as GDPR) or security constraints strictly prohibit the storage of historical user data.
This necessitates Exemplar-free CIL, where the model must adapt to new classes without any access to previous raw data. Traditional exemplar-free approaches often rely on weight regularization, which frequently fails to scale as the number of tasks increases. This research introduces the Self-distilled Knowledge Delegator (SKD), a generative framework that preserves the "essence" of learned knowledge by synthesizing informative samples rather than storing original data.
Methodology
The SKD framework operates in two distinct stages for every incremental step:
Stage One: Training the Delegator
The goal is to train a generator (the SKD) to produce synthetic data that captures the feature space of the previous model ($T_n$). This is achieved through a self-distillation mechanism involving two phases:
- Imitate Phase: A "naive" clone of the model ($S$) is trained to match the outputs of the old model ($T_n$) using the generated data.
- Explore Phase: This adversarial component pushes the SKD to generate data that causes the two models to disagree. By forcing the generator to find "hard" boundaries and diverse features, the system avoids mode collapse and ensures the synthetic data covers the full distribution of old classes.
- Helper BN Layer: To bridge the domain gap between synthetic noise and real-world data distributions, a specialized Batch Normalization layer is integrated into the generator.
Stage Two: Incremental Learning
In the second stage, the synthetic data produced by the SKD (representing old tasks) is combined with real data from the new task. The new model ($T_{n+1}$) is trained using a combination of standard Cross-Entropy loss and a Feature Cosine Discrepancy (FC-loss). The FC-loss ensures that the internal representations of old classes remain stable, preventing feature drift during the integration of new information.
Key Insights
- Feature Space over Image Quality: Unlike standard GANs designed to produce human-recognizable images, the SKD focuses on triggering the correct internal neural features. The synthetic samples may appear as noise to humans, but they are highly informative for the network's feature space.
- Adversarial Exploration: The "Explore" phase is critical. Without it, the generator would only produce "easy" samples that do not adequately represent the complex decision boundaries of the old model.
- The Helper BN Trick: Synthetic data often has statistics that differ significantly from real data. The Helper BN layer acts as a translator, allowing the model to process synthetic and real data simultaneously without performance degradation.
Experimental Results
The SKD method was evaluated on four benchmarks: CIFAR-100, ImageNet-Subset, Caltech-101, and Flowers-102.
- State-of-the-Art Performance: On the CIFAR-100 10-task benchmark, SKD achieved an average accuracy of 59.62%, outperforming the previous leading exemplar-free method (SDC+EWC) by 3.53%.
- Competitive with Exemplar-based Methods: Remarkably, SKD outperformed several methods that do use a memory buffer (e.g., iCaRL and BiC), demonstrating that high-quality synthetic delegation can be more effective than a limited set of real exemplars.
- Scalability: The method proved robust on high-resolution datasets like ImageNet-Subset, where previous data-free methods often struggled.
Conclusion and Future Work
The Self-distilled Knowledge Delegator provides a powerful, privacy-preserving alternative to exemplar-based incremental learning. By shifting the focus from data storage to knowledge delegation via synthetic feature-space exploration, SKD achieves state-of-the-art results in exemplar-free CIL. While performance still declines as the number of tasks grows due to compounding errors in the delegation chain, future research will focus on enhancing the long-term stability of the generator and applying these delegation principles to unsupervised domain adaptation.
Summary updated