Abstract: Online class-incremental continual learning (CL) studies the problem of learning new classes continually from an online non-stationary data stream, intending to adapt to new data while mitigating catastrophic forgetting. While memory replay has shown promising results, the recency bias in online learning caused by the commonly used Softmax classifier remains an unsolved challenge. Although the Nearest-Class-Mean (NCM) classifier is significantly undervalued in the CL community, we demonstrate that it is a simple yet effective substitute for the Softmax classifier. It addresses the recency bias and avoids structural changes in the fully-connected layer for new classes. Moreover, we observe considerable and consistent performance gains when replacing the Softmax classifier with the NCM classifier for several state-of-the-art replay methods. To leverage the NCM classifier more effectively, data embeddings belonging to the same class should be clustered and well-separated from those with a different class label. To this end, we contribute Supervised Contrastive Replay (SCR), which explicitly encourages samples from the same class to cluster tightly in embedding space while pushing those of different classes further apart during replay-based training. Overall, we observe that our proposed SCR substantially reduces catastrophic forgetting and outperforms state-of-the-art CL methods by a significant margin on a variety of datasets.
Introduction
In Online Class-Incremental Continual Learning, models must learn from a continuous stream of data where new classes appear over time. The primary challenge is catastrophic forgetting—the tendency for neural networks to overwrite old knowledge when learning new information. Most current replay-based methods rely on a Softmax classifier, which suffers from task-recency bias. Because the model encounters significantly more samples from the most recent task than from the memory buffer, the weights in the final fully-connected layer become heavily biased toward the newest data, causing the model to misclassify old samples as belonging to the most recent task.
Supervised Contrastive Replay (SCR)
To address these limitations, researchers from the University of Toronto and LG AI Research propose Supervised Contrastive Replay (SCR). SCR shifts the focus from learning classification weights to optimizing the geometry of the embedding space.
1. The Nearest-Class-Mean (NCM) Classifier
SCR replaces the standard Softmax layer with an NCM classifier. To classify an image, the model computes its embedding, finds the average embedding (prototype) for every class seen so far, and assigns the label of the nearest prototype. This approach is inherently more robust to class imbalance because it does not rely on learned weights that are easily skewed by the frequency of recent samples.
2. Supervised Contrastive Loss (SCL)
For NCM to be effective, embeddings must be tightly clustered by class. SCR utilizes Supervised Contrastive Loss (SCL) during training to explicitly pull embeddings of the same class together while pushing different classes apart.
3. Training and Inference
During training, the model processes a batch from the data stream alongside a batch from the memory buffer. It generates augmented views of these images and applies SCL to their representations. At test time, the projection head is discarded, and the model uses the current buffer samples to compute class means for the NCM classifier.
Experimental Results
SCR was evaluated on three standard benchmarks: Split CIFAR-10, Split CIFAR-100, and Split Mini-ImageNet, across various memory buffer sizes (M=200 to M=5000).
Performance Gains
SCR outperformed state-of-the-art (SOTA) methods like ER, MIR, GSS, and ASER by significant margins. For example, on Mini-ImageNet with a 1k memory buffer, SCR achieved 24.1% accuracy, nearly doubling the performance of the next best method, ASER (12.5%). On CIFAR-10, SCR reached 65.7% accuracy (M=1k), outperforming competitors by over 15 percentage points.
Embedding Quality
The effectiveness of SCR is visually evident in t-SNE visualizations. While standard methods like ER and MIR result in overlapping clusters, SCR produces highly distinct and well-separated class clusters.
Conclusion
SCR demonstrates that the Nearest-Class-Mean classifier is a powerful alternative to Softmax in continual learning settings. By combining NCM with Supervised Contrastive Loss, SCR effectively eliminates recency bias and achieves superior performance with high computational efficiency. This work highlights that optimizing the embedding space geometry is a critical path forward for robust online incremental learning.
Summary updated