NAPA-VQ: Neighborhood Aware Prototype Augmentation with Vector Quantization for Continual Learning

Tamasha Malepathirana, Damith A. Senanayake, Saman K. HalgamugeICCV 2023 · DOI · arXiv

Abstract

Non-Exemplar based Class Incremental Learning (NECIL) is a challenging paradigm where models must learn new classes without access to previous training data. We present NAPA-VQ, a framework that utilizes topological neighborhood information to mitigate catastrophic forgetting. By employing a Neighborhood-aware Vector Quantizer (NA-VQ) and a Neighborhood-aware Prototype Augmenter (NA-PA), the system maintains distinct class boundaries and generates high-fidelity synthetic prototypes. Our method achieves state-of-the-art results across several benchmarks, including CIFAR-100 and ImageNet-1K.

1. Introduction

Deep neural networks typically struggle with continual learning, often overwriting previously acquired knowledge when trained on new tasks—a phenomenon known as catastrophic forgetting. While exemplar-based methods mitigate this by storing a subset of old data, privacy concerns and memory constraints often make this impractical. Non-Exemplar based Class Incremental Learning (NECIL) addresses these constraints but faces the challenge of overlapping feature representations between old and new classes. NAPA-VQ addresses this by explicitly modeling the topological relationships between classes in the feature space.

2. Methodology

The NAPA-VQ framework consists of two primary components designed to preserve and refine the model's internal representation of class boundaries.

2.1 Neighborhood-aware Vector Quantizer (NA-VQ)

Inspired by "Neural Gas" networks, the NA-VQ module learns the topology of the feature space. It identifies "neighboring" classes—those most likely to be confused by the model. The module applies attractive forces to pull data points toward their respective class centers (Coding Vectors) and repulsive forces to push them away from confusing neighbors. This process ensures that the feature space remains organized and that decision boundaries are clearly defined. The training is guided by a loss function comprising Knowledge Distillation ($L_{KD}$), Neighborhood-aware loss ($L_{NA}$), and Decoupled Cross-Entropy ($L_{DCE}$). A connectivity factor of $K=15$ was found to optimally balance performance and efficiency.

2.2 Neighborhood-aware Prototype Augmenter (NA-PA)

To compensate for the lack of original data, NAPA-VQ utilizes synthetic prototypes. Unlike standard methods that simply add noise to class means, the NA-PA module generates new prototypes by fusing an old class mean ($μ_i$) with the means of its neighbors ($μ_j$). This fusion is controlled by a parameter $α \in N(0.5, 1)$, effectively populating the most uncertain regions of the feature space. This "neighborhood-aware" rehearsal allows the model to maintain sharp boundaries even as new classes are introduced.

The NAPA-VQ framework architecture, showing the integration of NA-VQ and NA-PA modules.

3. Experimental Evaluation

The performance of NAPA-VQ was evaluated on four standard computer vision benchmarks: CIFAR-100, TinyImageNet, ImageNet-Subset, and the full-scale ImageNet-1K. The experiments followed incremental learning protocols with 5, 10, and 20 tasks to assess the model's stability over time. All experiments utilized a fixed-size ResNet-18 backbone.

4. Results and Discussion

NAPA-VQ consistently outperformed existing state-of-the-art (SOTA) methods, including PASS, IL2A, and SSRE.

  • Accuracy Improvements: The method showed an average accuracy increase of 5% on CIFAR-100, 2% on TinyImageNet, and 4% on ImageNet-Subset.
  • Mitigation of Forgetting: NAPA-VQ drastically reduced catastrophic forgetting, achieving a 10% improvement on CIFAR-100 compared to previous non-exemplar methods.
  • Scalability: The framework maintained robust performance on ImageNet-1K, demonstrating its applicability to large-scale real-world problems.
Performance comparison of NAPA-VQ against state-of-the-art methods on CIFAR-100 and TinyImageNet.

The success of NAPA-VQ highlights the importance of class topology. By treating classes as part of a neighborhood rather than isolated entities, the model can focus its learning on the most critical decision boundaries.

5. Conclusion

NAPA-VQ represents a significant advancement in privacy-preserving continual learning. By combining topological mapping with smart prototype augmentation, it effectively counters catastrophic forgetting in NECIL scenarios. Future work will explore using multiple coding vectors per class to capture more complex distributions and exploring how this framework integrates with dynamically expanding networks.

Summary updated

BibTeX
@inproceedings{malepathirana2023napa,
  author = {Tamasha Malepathirana and Damith A. Senanayake and Saman K. Halgamuge},
  title = {{NAPA-VQ: Neighborhood Aware Prototype Augmentation with Vector Quantization for Continual Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01072},
}
Figure