Enhancing Generative Class Incremental Learning Performance With a Model Forgetting Approach

Taro Togo, Ren Togo, Keisuke Maeda, Takahiro OgawaIEEE Open Journal of Signal Processing 2025 · DOI

Abstract

In the field of deep learning, the ability to learn new information sequentially without degrading previously acquired knowledge—known as Continual Learning—remains a significant challenge. This paper investigates a novel approach to Generative Class Incremental Learning (GCIL) inspired by biological processes: intentional forgetting. By selectively unlearning specific classes before introducing new ones, we demonstrate that a model can reduce concept leakage and more effectively allocate its representational capacity. Our experiments using Variational Autoencoders (VAEs) on MNIST and Fashion-MNIST benchmarks show that this "Forget then Learn" strategy significantly improves the acquisition of new classes while preserving the integrity of retained information.

1. Introduction

Deep learning models are typically static; once trained, they struggle to incorporate new data without undergoing full retraining. When a pre-trained model is fine-tuned on new categories, it often suffers from catastrophic forgetting, where the weights optimized for previous tasks are overwritten, leading to a total loss of prior knowledge.

This issue is particularly acute in Generative Class Incremental Learning (GCIL). In GCIL, the objective is to enable a generative model to produce high-quality samples of new categories while maintaining its ability to generate samples from earlier classes. While most research focuses on preservation mechanisms, this study explores whether strategic unlearning can actually facilitate the learning of new information, mimicking the human brain's ability to discard irrelevant data to make room for new concepts.

2. Methodology

The proposed framework follows a two-stage workflow: Selective Forgetting followed by Incremental Learning.

2.1 Selective Forgetting

The researchers utilize a technique termed Selective Amnesia (SA). Rather than simply pruning weights, the model is retrained to dissociate a specific "to-be-forgotten" class ($c_f$) from its original features. This is achieved by mapping the class embedding to meaningless information, such as white noise or—more effectively—the embedding of the upcoming new class ($c_{new}$). This process effectively "resets" the specific region of the representation space previously occupied by the forgotten class.

2.2 Incremental Learning

Once the target class is unlearned, the new class ($c_{new}$) is introduced. To prevent the model from forgetting the remaining classes ($c_r$), the authors employ Elastic Weight Consolidation (EWC). EWC acts as a regularizer that penalizes changes to weights identified as critical for the retained classes, effectively anchoring the model's core knowledge while allowing flexibility for the new task.

3. Experimental Setup

The experiments were conducted using a one-hot Variational Autoencoder (VAE) architecture. The approach was evaluated on two standard benchmarks:

  • MNIST: Handwritten digits (0-9).
  • Fashion-MNIST: Clothing items (e.g., T-shirts, trousers, boots).

In the experimental protocol, the model is initially trained on 9 classes. It is then tasked with "forgetting" one specific class before learning a 10th new class. This setup allows for a precise measurement of how the transition between classes is managed by the forgetting mechanism.

4. Key Insights and Results

4.1 Forgetting as a Catalyst

The study finds that intentional unlearning serves as a catalyst for new learning by reducing "concept leakage." By clearing the representation of an old class, the model encounters less interference when mapping the features of a new, potentially similar class.

4.2 The Role of "Pseudo-Memory"

The authors discovered that the replacement target for the forgotten class matters. Replacing the forgotten class with the embedding of the new class (Pseudo-Memory) before it is fully learned yielded superior results compared to using random white noise. This suggests that pre-allocating the space for future knowledge is more efficient than leaving it empty.

4.3 Performance Metrics

Using an external classifier to evaluate the generative output, the researchers observed:

  • Accuracy Gains: In MNIST trials, adding the forgetting mechanism (PM) to standard fine-tuning or EWC significantly boosted the classification accuracy of the new class. In specific configurations, accuracy rose from 0.35 to 0.55.
  • Effective Erasure: The classification probability for "forgotten" classes dropped to near-random levels (below 0.10), confirming successful unlearning.
  • Stability: Accuracy for retained classes remained high (typically above 0.90 for MNIST), proving that the forgetting was indeed selective and did not cause collateral damage to the rest of the model's knowledge base.

5. Discussion and Limitations

While the results are promising, the study identifies several areas for future exploration. The current experiments were limited to relatively simple, low-resolution datasets (MNIST and Fashion-MNIST). Whether these findings scale to high-resolution images or more complex architectures like Diffusion models and Transformers remains to be seen.

Furthermore, the current process requires a human to specify which class should be forgotten. A more robust system would involve automated "capacity management," where the model autonomously identifies obsolete or redundant information to discard as it encounters new data.

6. Conclusion

This research demonstrates that forgetting is not merely a failure of memory but can be a strategic tool in machine learning. By intentionally clearing out specific representational spaces, models can overcome the limitations of static capacity and learn new tasks more effectively. This "Forget then Learn" paradigm offers a compelling new direction for the development of truly continual AI systems.

Summary updated

Abstract

This study presents a novel approach to Generative Class Incremental Learning (GCIL) by introducing the forgetting mechanism, aimed at dynamically managing class information for better adaptation to streaming data. GCIL is one of the hot topics in the field of computer vision, and it is considered one of the important tasks in society as one of the continual learning approaches for generative models. The ability to forget is a crucial brain function that facilitates continual learning by selectively discarding less relevant information for humans. However, in the field of machine learning models, the concept of intentionally forgetting has not been extensively investigated. In this study, we aim to bridge this gap by incorporating the forgetting mechanisms into GCIL, thereby examining their impact on the models' ability to learn in continual learning. Through our experiments, we have found that integrating the forgetting mechanisms significantly enhances the models' performance in acquiring new knowledge, underscoring the positive role that strategic forgetting plays in the process of continual learning.
BibTeX
@article{togo2025enhancing,
  author = {Taro Togo and Ren Togo and Keisuke Maeda and Takahiro Ogawa and Ren Togo},
  title = {{Enhancing Generative Class Incremental Learning Performance With a Model Forgetting Approach}},
  journal = {IEEE Open Journal of Signal Processing},
  year = {2025},
  doi = {10.1109/ojsp.2025.3528900},
}

Loading related content...

Figure