Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning

Zheda Mai, Ruiwen Li, Hyunwoo Kim, Scott Sanner

Abstract

Online class-incremental continual learning (CL) studies the problem of learning new classes continually from an online non-stationary data stream, intending to adapt to new data while mitigating catastrophic forgetting. While memory replay has shown promising results, the recency bias in online learning caused by the commonly used Softmax classifier remains an unsolved challenge. Although the Nearest-Class-Mean (NCM) classifier is significantly undervalued in the CL community, we demonstrate that it is a simple yet effective substitute for the Softmax classifier. It addresses the recency bias and avoids structural changes in the fully-connected layer for new classes. Moreover, we observe considerable and consistent performance gains when replacing the Softmax classifier with the NCM classifier for several state-of-the-art replay methods. To leverage the NCM classifier more effectively, data embeddings belonging to the same class should be clustered and well-separated from those with a different class label. To this end, we contribute Supervised Contrastive Replay (SCR), which explicitly encourages samples from the same class to cluster tightly in embedding space while pushing those of different classes further apart during replay-based training. Overall, we observe that our proposed SCR substantially reduces catastrophic forgetting and outperforms state-of-the-art CL methods by a significant margin on a variety of datasets.

Introduction

In Online Class-Incremental Continual Learning, a model must learn new classes from a continuous, one-pass stream of data. The primary obstacle is catastrophic forgetting, often exacerbated by the "recency bias" of standard Softmax classifiers. Because the model encounters many samples of new classes but only a few replayed samples of old ones, the final fully-connected layer's weights become heavily biased toward the newest data. This causes the model to misclassify old samples even if the feature extractor still recognizes them.

The SCR Framework

Supervised Contrastive Replay (SCR) addresses these issues by fundamentally changing the classification paradigm. Instead of a learnable linear layer, SCR employs a Nearest Class Mean (NCM) classifier.

Overview of the Supervised Contrastive Replay (SCR) training and testing phases.

Nearest Class Mean (NCM) Classifier

During inference, the NCM classifier calculates the average embedding (mean) for every class seen so far using samples stored in a memory buffer. A test image is classified by finding the class mean closest to its embedding in the feature space. This approach is inherently more robust to class imbalance than Softmax because it relies on the geometric center of class clusters rather than biased weight magnitudes.

Supervised Contrastive Loss (SCL)

To maximize the effectiveness of NCM, the embedding space must be highly structured. SCR utilizes a Supervised Contrastive Loss during training. For each minibatch—composed of new streaming data and replayed buffer data—the model generates multiple augmented views. The SCL pulls embeddings of the same class together while pushing those of different classes apart. This ensures that class clusters are tight and well-separated, which is the ideal condition for a distance-based classifier like NCM.

Experimental Results

The authors evaluated SCR on Split CIFAR-10, Split CIFAR-100, and Split Mini-ImageNet. SCR consistently outperformed state-of-the-art (SOTA) methods across all datasets and buffer sizes. For instance, on CIFAR-100 with a 2,000-sample buffer, SCR achieved 32.8% average accuracy, a significant jump over the next best SOTA method (ASER-NCM) at 25.2%.

Qualitative Analysis

The impact of SCR on the embedding space is clearly visible through t-SNE visualizations.

t-SNE visualizations comparing embedding clusters of ER, MIR, ASER, and SCR.

As shown above, standard methods like Experience Replay (ER) and Maximally Interfered Retrieval (MIR) result in overlapping and poorly defined clusters. In contrast, SCR produces distinct, compact clusters for each class. This structural clarity is what enables the NCM classifier to achieve superior performance and mitigate the recency bias that plagues traditional architectures.

Conclusion

SCR demonstrates that the NCM classifier is a powerful yet overlooked tool for continual learning. By combining NCM with a supervised contrastive loss, SCR creates a robust framework that avoids the architectural rigidity and bias of Softmax classifiers. The result is a simple, efficient, and highly effective method for online class-incremental learning.

Summary updated

Abstract

Online class-incremental continual learning (CL) studies the problem of learning new classes continually from an online non-stationary data stream, intending to adapt to new data while mitigating catastrophic forgetting. While memory replay has shown promising results, the recency bias in online learning caused by the commonly used Softmax classifier remains an unsolved challenge. Although the Nearest-Class-Mean (NCM) classifier is significantly undervalued in the CL community, we demonstrate that it is a simple yet effective substitute for the Softmax classifier. It addresses the recency bias and avoids structural changes in the fully-connected layer for new classes. Moreover, we observe considerable and consistent performance gains when replacing the Softmax classifier with the NCM classifier for several state-of-the-art replay methods. To leverage the NCM classifier more effectively, data embeddings belonging to the same class should be clustered and well-separated from those with a different class label. To this end, we contribute Supervised Contrastive Replay (SCR), which explicitly encourages samples from the same class to cluster tightly in embedding space while pushing those of different classes further apart during replay-based training. Overall, we observe that our proposed SCR substantially reduces catastrophic forgetting and outperforms state-of-the-art CL methods by a significant margin on a variety of datasets.
BibTeX
@inproceedings{mai2021supervised,
  author = {Zheda Mai and Ruiwen Li and Hyunwoo Kim and Scott Sanner},
  title = {{Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=qv1174nusiA},
}
Figure