Leveraging Self-Supervised and Supervised Embeddings for Memory-Efficient Experience-Replay Continual Learning
Lists
Only you see this
Abstract
Catastrophic forgetting remains a key challenge in Continual Learning (CL). In replay-based CL with severe memory constraints, performance critically depends on the sample selection strategy - that is, which examples are stored for replay. Most existing approaches construct memory buffers using embeddings learned under supervised objectives. However, class-agnostic, self-supervised representations often encode rich, class-relevant semantics that are overlooked. We propose a new method, MERS - Multiple Embedding Replay Selection, which replaces the buffer selection module with a graph-based approach that integrates both supervised and self-supervised embeddings. Empirical results show consistent improvements over state-of-the-art selection strategies across a range of continual learning algorithms, with particularly strong gains in low-memory regimes. On CIFAR-100 and TinyImageNet, MERS outperforms single-embedding baselines without adding model parameters or increasing replay volume, making it a practical, drop-in enhancement for replay-based continual learning.
BibTeX
@misc{yanowsky2026leveraging,
author = {Danit Yanowsky and Daphna Weinshall},
title = {{Leveraging Self-Supervised and Supervised Embeddings for Memory-Efficient Experience-Replay Continual Learning}},
year = {2026},
url = {https://openreview.net/forum?id=7xqqptZP0x},
howpublished = {OpenReview},
note = {Submitted to ICLR 2026},
}