Abstract
Non-Exemplar based Class Incremental Learning (NECIL) is a challenging paradigm where models must learn new classes without access to previous training data. We present NAPA-VQ, a framework that utilizes topological neighborhood information to mitigate catastrophic forgetting. By employing a Neighborhood-aware Vector Quantizer (NA-VQ) and a Neighborhood-aware Prototype Augmenter (NA-PA), the system maintains distinct class boundaries and generates high-fidelity synthetic prototypes. Our method achieves state-of-the-art results across several benchmarks, including CIFAR-100 and ImageNet-1K.
1. Introduction
Deep neural networks typically struggle with continual learning, often overwriting previously acquired knowledge when trained on new tasks—a phenomenon known as catastrophic forgetting. While exemplar-based methods mitigate this by storing a subset of old data, privacy concerns and memory constraints often make this impractical. Non-Exemplar based Class Incremental Learning (NECIL) addresses these constraints but faces the challenge of overlapping feature representations between old and new classes. NAPA-VQ addresses this by explicitly modeling the topological relationships between classes in the feature space.
2. Methodology
The NAPA-VQ framework consists of two primary components designed to preserve and refine the model's internal representation of class boundaries.
2.1 Neighborhood-aware Vector Quantizer (NA-VQ)
Inspired by "Neural Gas" networks, the NA-VQ module learns the topology of the feature space. It identifies "neighboring" classes—those most likely to be confused by the model. The module applies attractive forces to pull data points toward their respective class centers (Coding Vectors) and repulsive forces to push them away from confusing neighbors. This process ensures that the feature space remains organized and that decision boundaries are clearly defined. The training is guided by a loss function comprising Knowledge Distillation ($L_{KD}$), Neighborhood-aware loss ($L_{NA}$), and Decoupled Cross-Entropy ($L_{DCE}$). A connectivity factor of $K=15$ was found to optimally balance performance and efficiency.
2.2 Neighborhood-aware Prototype Augmenter (NA-PA)
To compensate for the lack of original data, NAPA-VQ utilizes synthetic prototypes. Unlike standard methods that simply add noise to class means, the NA-PA module generates new prototypes by fusing an old class mean ($μ_i$) with the means of its neighbors ($μ_j$). This fusion is controlled by a parameter $α \in N(0.5, 1)$, effectively populating the most uncertain regions of the feature space. This "neighborhood-aware" rehearsal allows the model to maintain sharp boundaries even as new classes are introduced.
3. Experimental Evaluation
The performance of NAPA-VQ was evaluated on four standard computer vision benchmarks: CIFAR-100, TinyImageNet, ImageNet-Subset, and the full-scale ImageNet-1K. The experiments followed incremental learning protocols with 5, 10, and 20 tasks to assess the model's stability over time. All experiments utilized a fixed-size ResNet-18 backbone.
4. Results and Discussion
NAPA-VQ consistently outperformed existing state-of-the-art (SOTA) methods, including PASS, IL2A, and SSRE.
- Accuracy Improvements: The method showed an average accuracy increase of 5% on CIFAR-100, 2% on TinyImageNet, and 4% on ImageNet-Subset.
- Mitigation of Forgetting: NAPA-VQ drastically reduced catastrophic forgetting, achieving a 10% improvement on CIFAR-100 compared to previous non-exemplar methods.
- Scalability: The framework maintained robust performance on ImageNet-1K, demonstrating its applicability to large-scale real-world problems.
The success of NAPA-VQ highlights the importance of class topology. By treating classes as part of a neighborhood rather than isolated entities, the model can focus its learning on the most critical decision boundaries.
5. Conclusion
NAPA-VQ represents a significant advancement in privacy-preserving continual learning. By combining topological mapping with smart prototype augmentation, it effectively counters catastrophic forgetting in NECIL scenarios. Future work will explore using multiple coding vectors per class to capture more complex distributions and exploring how this framework integrates with dynamically expanding networks.
Summary updated