Abstract
Recently, because of the high-quality representations of contrastive learning methods, rehearsal-based contrastive continual learning has been proposed to explore how to continually learn transferable representation embeddings to avoid the catastrophic forgetting issue in traditional continual settings. Based on this framework, we propose Contrastive Continual Learning via Importance Sampling (CCLIS) to preserve knowledge by recovering previous data distributions with a new strategy for Replay Buffer Selection (RBS), which minimize estimated variance to save hard negative samples for representation learning with high quality. Furthermore, we present the Prototype-instance Relation Distillation (PRD) loss, a technique designed to maintain the relationship between prototypes and sample representations using a self-distillation process. Experiments on standard continual learning benchmarks reveal that our method notably outperforms existing baselines in terms of knowledge preservation and thereby effectively counteracts catastrophic forgetting in online contexts.
Introduction
Deep neural networks are notoriously "forgetful." When trained on a sequence of tasks (e.g., first learning to recognize cats, then birds), they often lose the ability to perform the first task—a phenomenon known as Catastrophic Forgetting.
To fight this, researchers use Continual Learning (CL). A popular approach is "rehearsal," where a tiny "replay buffer" of old images is kept to remind the model of what it saw before. Recently, Contrastive Learning—which teaches models by comparing similar and different images—has shown promise in creating robust features. However, two major gaps remain: distribution bias due to small buffer sizes and the difficulty of selecting the most informative "hard negative" samples for representation learning within a limited memory budget.
Methods
The authors propose CCLIS (Contrastive Continual Learning via Importance Sampling), which rests on three innovative pillars:
- Importance Sampling (IS): Instead of treating all buffered samples equally, the model assigns "weights" to them. This mathematically approximates the full data distribution of previous tasks, effectively "recovering" the lost context of the past without needing more memory.
- Replay Buffer Selection (RBS): To make the Importance Sampling effective, the authors developed a strategy to minimize estimation variance. They discovered that the best samples to keep are those that are difficult for the model to distinguish from other classes on average—essentially automated hard negative mining.
- Prototype-Instance Relation Distillation (PRD): The model maintains "prototypes" (representative centers for each class). The PRD loss ensures that the relationship (similarity) between a specific image and these class centers remains consistent even as the model updates. This acts as an anchor, preventing the model's "internal map" from warping too much when new tasks arrive.
Experimental Results
The authors evaluated their method on three standard benchmarks: Seq-Cifar-10, Seq-Cifar-100, and Seq-Tiny-ImageNet. They tested these across Class-Incremental and Task-Incremental scenarios.
CCLIS consistently outperformed state-of-the-art baselines (like Co2L, DER++, and GCR) across all datasets:
- Accuracy Boost: On Seq-Cifar-10 (Class-IL) with a 200-image buffer, CCLIS achieved 74.95% accuracy, outperforming the previous best (Co2L) by nearly 9.4%.
- Reduced Forgetting: The "Average Forgetting" metric was significantly lower. For instance, on Seq-Cifar-10, CCLIS showed only 22.59% forgetting compared to 36.35% for Co2L.
As shown in the ablation studies, the distillation power (λ) significantly impacts performance. The test accuracy peaks around λ=0.6, demonstrating that maintaining prototype-instance relations is crucial for stabilizing the model's memory across tasks.
Conclusion
CCLIS addresses the core challenges of contrastive continual learning by intelligently recovering past data distributions and stabilizing feature representations. By linking variance minimization to hard negative selection and using prototypes as anchors, the framework achieves state-of-the-art performance with minimal memory overhead. Future work aims to develop even more sophisticated algorithms to recover the entire distribution history and further bridge the gap between online continual learning and traditional offline training.
Summary updated