Abstract
Recently, because of the high-quality representations of contrastive learning methods, rehearsal-based contrastive continual learning has been proposed to explore how to continually learn transferable representation embeddings to avoid the catastrophic forgetting issue in traditional continual settings. Based on this framework, we propose Contrastive Continual Learning via Importance Sampling (CCLIS) to preserve knowledge by recovering previous data distributions with a new strategy for Replay Buffer Selection (RBS), which minimize estimated variance to save hard negative samples for representation learning with high quality. Furthermore, we present the Prototype-instance Relation Distillation (PRD) loss, a technique designed to maintain the relationship between prototypes and sample representations using a self-distillation process. Experiments on standard continual learning benchmarks reveal that our method notably outperforms existing baselines in terms of knowledge preservation and thereby effectively counteracts catastrophic forgetting in online contexts.
Introduction
Deep learning models are notoriously susceptible to Catastrophic Forgetting (CF), where training on new tasks leads to a rapid decline in performance on previously learned ones. Continual Learning (CL) seeks to mitigate this, frequently employing rehearsal-based strategies that maintain a small buffer of past data to "remind" the model of previous tasks.
Contrastive learning has recently emerged as a powerful tool for CL due to its ability to learn robust, high-quality representations. However, its application in a streaming, online setting is hindered by two main issues: Distribution Bias, where the discrepancy between the full offline dataset and the sparse online buffer leads to biased representations; and Hard Negative Selection, as identifying the most informative "hard" samples for future, unknown tasks is a significant challenge in a sequential learning environment.
Methodology
The proposed framework, Contrastive Continual Learning via Importance Sampling (CCLIS), addresses these challenges through three integrated components:
1. Importance Sampling (CCLIS)
To bridge the gap between online and offline training, the model employs importance sampling. This technique estimates the original data distribution of previous tasks, effectively "recovering" the lost distribution from the limited buffered samples and narrowing the gap between online and offline training regimes.
2. Replay Buffer Selection (RBS)
The authors introduce a novel strategy for selecting which samples to retain in the memory buffer. By minimizing the estimated variance of the importance weights, the RBS strategy naturally prioritizes hard negative samples, which are the most informative for maintaining discriminative feature boundaries.
3. Prototype-Instance Relation Distillation (PRD)
To ensure stability as the model evolves, the Prototype-instance Relation Distillation (PRD) loss is introduced. This self-distillation process maintains the relative relationships between class "prototypes" (centroid representations) and individual instances, preventing the feature drift that typically leads to forgetting.
Experimental Evaluation
The method was rigorously tested on three standard benchmarks: Seq-Cifar-10, Seq-Cifar-100, and Seq-Tiny-ImageNet. These datasets present a sequential learning challenge where classes are introduced in stages, making them rigorous tests for catastrophic forgetting.
Key Results
CCLIS demonstrates superior performance compared to state-of-the-art baselines like Co2L and DER++:
- Accuracy: On Seq-Cifar-10 with a 200-sample buffer, CCLIS achieved 74.95% accuracy, outperforming the previous best contrastive method (Co2L) by nearly 10%.
- Forgetting: The method significantly reduced "Average Forgetting." For instance, on Seq-Cifar-100, forgetting was limited to 46.89%, compared to 62.72% for DER.
- Ablation Studies: Results confirmed that both Importance Sampling and the PRD loss are critical; removing either component led to a substantial drop in performance.
Key Insights and Discussion
A qualitative analysis using t-SNE visualizations reveals that the RBS method effectively selects samples located at the "edges" of class clusters. These boundary samples serve as the most effective hard negatives, providing the necessary contrast to maintain clear separations between classes as new tasks are added. The core mathematical insight is that the effectiveness of importance sampling depends on minimizing variance; by picking samples that minimize this variance, the model automatically selects the most informative data points for the buffer.
Conclusion
CCLIS effectively counters catastrophic forgetting by combining distribution recovery via importance sampling with relational stability through distillation. While the method significantly improves knowledge preservation, it is currently limited by its reliance on the last visible samples to reconstruct distributions. Future research will explore more robust ways to recover historical data distributions as the memory of early tasks becomes increasingly sparse over long training sequences.
The code is available at https://github.com/lijy373/CCLIS.
Summary updated