Introduction
Class-Incremental Learning (CIL) is a fundamental challenge in AI, requiring models to learn new classes sequentially without degrading performance on previously learned ones. This phenomenon, known as Catastrophic Forgetting, occurs because model parameters shift to accommodate new data, often at the expense of old representations. While leveraging Pre-Trained Models (PTMs) like Vision Transformers (ViT) offers a strong starting point, challenges such as the domain gap between pre-training and downstream tasks, and suboptimal class relationships in sequential learning, persist.
The NC-PTM Framework
To address these challenges, the authors introduce NC-PTM (Neural Collapse for Pre-Trained Models). This framework utilizes Neural Collapse (NC) theory to guide the model toward a geometrically optimal feature distribution.
Core Components
- ETF Classifier: Instead of learning a classifier, the model uses a pre-assigned, frozen Equiangular Tight Frame (ETF). An ETF consists of vectors maximally separated by equal angles, providing a fixed "North Star" for class features.
- Task-Related Adaptation (TRA): Lightweight adapters are inserted into the transformer blocks of the PTM. Only these adapters are tuned, allowing the model to specialize for new tasks without destroying the generalized knowledge of the backbone.
- Feature Compression Module (FCM) & Optimal Structural Alignment (OSA): The FCM sits between the backbone and the classifier, tasked with "compressing" features into a single point—the corresponding ETF weight. The OSA loss ensures that within-class variability collapses ($NC_1$) and class means converge to the ETF anchors ($NC_2$).
Experimental Evaluation
The method was evaluated on seven diverse benchmarks, including standard datasets like CIFAR-100 and robustness-focused ones like ImageNet-R, ImageNet-A, and ObjectNet.
Key Results
- Superior Accuracy: NC-PTM consistently outperformed state-of-the-art (SOTA) methods. Notably, on ImageNet-R, it achieved an average incremental accuracy of 81.64%, representing a 3.33% improvement over the previous best method.
- Geometric Stability: By minimizing within-class variance (the "Compress to One Point" approach), the model creates more "room" in the feature space, significantly reducing the likelihood of overlap between old and new classes.
- Memory Efficiency: The "Class Memory" (OUR-CM) version achieves competitive results without storing raw images or features of old classes, offering a scalable solution for memory-constrained applications.
Conclusion and Future Work
NC-PTM demonstrates that pre-defining the geometric structure of the feature space through Neural Collapse can effectively mitigate catastrophic forgetting in incremental learning. While the "Full Memory" (OUR-FM) version currently offers the best performance, future research will focus on scaling these principles to open-ended systems and extending the paradigm to Large Language Models (LLMs) and multi-modal tasks.
Summary updated