Leveraging Self-Supervised and Supervised Embeddings for Memory-Efficient Experience-Replay Continual Learning

Danit Yanowsky, Daphna WeinshallOpenReview 2026 · Submission to ICLR 2026 · Rejected · OpenReview

Abstract

Catastrophic forgetting remains a key challenge in Continual Learning (CL). In replay-based CL with severe memory constraints, performance critically depends on the sample selection strategy - that is, which examples are stored for replay. Most existing approaches construct memory buffers using embeddings learned under supervised objectives. However, class-agnostic, self-supervised representations often encode rich, class-relevant semantics that are overlooked. We propose a new method, MERS - Multiple Embedding Replay Selection, which replaces the buffer selection module with a graph-based approach that integrates both supervised and self-supervised embeddings. Empirical results show consistent improvements over state-of-the-art selection strategies across a range of continual learning algorithms, with particularly strong gains in low-memory regimes. On CIFAR-100 and TinyImageNet, MERS outperforms single-embedding baselines without adding model parameters or increasing replay volume, making it a practical, drop-in enhancement for replay-based continual learning.
BibTeX
@misc{yanowsky2026leveraging,
  author = {Danit Yanowsky and Daphna Weinshall},
  title = {{Leveraging Self-Supervised and Supervised Embeddings for Memory-Efficient Experience-Replay Continual Learning}},
  year = {2026},
  url = {https://openreview.net/forum?id=7xqqptZP0x},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2026},
}
Figure