Incremental Learning Using Conditional Adversarial Networks

Ye Xiang, Ying Fu, Pan Ji, Hua Huang

Abstract

Deep Neural Networks face significant challenges in sequential learning, primarily due to catastrophic forgetting. We present a novel framework that utilizes Conditional Generative Adversarial Networks (cGANs) to mitigate this issue. Unlike traditional methods that store raw image exemplars, our approach preserves only the statistical distributions (mean and covariance) of class-specific feature embeddings. A generator then produces synthetic "pseudo-features" to facilitate adversarial rehearsal during incremental steps. Evaluations on CIFAR-100 and CUB-200 show that our method maintains higher accuracy and better plasticity than current state-of-the-art solutions while remaining memory-efficient.

Introduction

The ability to learn new information incrementally without forgetting previously acquired knowledge is a hallmark of biological intelligence but remains a hurdle for artificial neural networks. When trained on new data, standard models often experience "catastrophic forgetting," where weights optimized for previous tasks are overwritten. Existing solutions typically involve storing a subset of original images (exemplar-based) or freezing parts of the network (architecture-based). However, these methods either demand excessive memory or restrict the model's ability to learn complex new features.

Proposed Framework

The proposed architecture balances memory efficiency with model plasticity by operating in the feature space rather than the pixel space.

The proposed framework consists of a fixed Base Sub-net (B), a Generator (G) for pseudo-features, and a dual-headed Discriminator (D).

The system comprises three primary modules:

  1. Base Sub-net ($B$): A pre-trained feature extractor (e.g., ResNet) that remains fixed after the initial training phase to provide stable embeddings.
  2. Generator ($G$): This component generates synthetic feature maps ("pseudo-examples") conditioned on class-specific embeddings. These embeddings are sampled from Gaussian distributions defined by the stored mean and covariance of previous classes.
  3. Discriminator ($D$): A dual-headed network that performs both multi-class classification and adversarial discrimination (real vs. synthetic).

During incremental learning, the model combines real features from new classes with pseudo-features generated from the stored statistics of old classes. This "adversarial rehearsal" allows the discriminator to update its classification boundaries for both old and new categories simultaneously.

Experimental Results

The framework was evaluated on the CIFAR-100 and CUB-200 datasets, simulating incremental learning by introducing classes in varying batch sizes (steps of 2, 5, 10, or 25).

Accuracy comparison on CIFAR-100 and CUB-200. The proposed method (Ours) exhibits significantly slower performance decay compared to baselines like iCaRL and FearNet.

As shown in the performance plots, the proposed method consistently outperforms state-of-the-art baselines. While competitors like iCaRL and FearNet show rapid accuracy drops as the number of classes increases, our approach maintains a more robust performance profile.

Key insights from the experiments include:

  • Efficiency: Storing second-order statistics (mean and covariance) requires constant memory per class, $O(d^2)$, which is far more scalable than storing raw images.
  • Ablation Studies: Removing the adversarial loss led to a significant decrease in the retention of old knowledge, confirming that the "real vs. fake" discrimination is vital for high-quality pseudo-feature generation.
  • Reconstruction Loss: Implementing a cosine-distance-based reconstruction loss prevents the generator from drifting, ensuring that synthetic features remain representative of their target classes.

Conclusion

This work demonstrates that generating perceptual features via cGANs is a powerful alternative to storing raw data for incremental learning. By leveraging statistical compression and adversarial rehearsal, the model achieves a superior balance between stability and plasticity. Future research will explore relaxing the Gaussian distribution assumption and adapting the base sub-net to further improve feature extraction for highly diverse sequential tasks.

Summary updated

Abstract

Incremental learning using Deep Neural Networks (DNNs) suffers from catastrophic forgetting. Existing methods mitigate it by either storing old image examples or only updating a few fully connected layers of DNNs, which, however, requires large memory footprints or hurts the plasticity of models. In this paper, we propose a new incremental learning strategy based on conditional adversarial networks. Our new strategy allows us to use memory-efficient statistical information to store old knowledge, and fine-tune both convolutional layers and fully connected layers to consolidate new knowledge. Specifically, we propose a model consisting of three parts, i.e., a base sub-net, a generator, and a discriminator. The base sub-net works as a feature extractor which can be pre-trained on large scale datasets and shared across multiple image recognition tasks. The generator conditioned on labeled embeddings aims to construct pseudo-examples with the same distribution as the old data. The discriminator combines real-examples from new data and pseudo-examples generated from the old data distribution to learn representation for both old and new classes. Through adversarial training of the discriminator and generator, we accomplish the multiple continuous incremental learning. Comparison with the state-of-the-arts on public CIFAR-100 and CUB-200 datasets shows that our method achieves the best accuracies on both old and new classes while requiring relatively less memory storage.
BibTeX
@inproceedings{xiang2019incremental,
  author = {Ye Xiang and Ying Fu and Pan Ji and Hua Huang},
  title = {{Incremental Learning Using Conditional Adversarial Networks}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00672},
}
Figure