Replay Can Provably Increase Forgetting

Yasaman Mahdaviyeh, James Lucas, Mengye Ren, Andreas S. Tolias, Richard S. Zemel, Toniann Pitassi

Abstract

Experience replay is widely considered the "gold standard" for mitigating catastrophic forgetting in continual learning. However, this research demonstrates that replay can be a double-edged sword. Through theoretical analysis of over-parameterized linear regression and empirical validation with neural networks, we show that replaying a small number of samples can paradoxically increase forgetting compared to no replay at all.

Introduction

When machine learning models learn tasks sequentially, they often suffer from catastrophic forgetting: mastering a new task (Task B) leads to the loss of performance on a previous one (Task A). The standard remedy is experience replay, where a buffer of old data is mixed into current training. While generally effective, the theoretical foundations of replay are limited. This paper challenges the assumption that "some replay is always better than none," identifying specific scenarios where replay is counterproductive.

Methodology

The study approaches the problem through two distinct lenses:

  1. Theoretical Analysis: The authors examine over-parameterized linear regression. While simple enough for closed-form mathematical solutions via orthogonal projections, this setting is complex enough to exhibit catastrophic forgetting. Analysis covers both worst-case task sequences and average-case scenarios using random task subspaces.
  2. Empirical Validation: To ensure findings extend beyond linear models, Multi-Layer Perceptrons (MLPs) were tested using standard optimizers (SGD and Adam) on classic benchmarks like Rotated MNIST and Split MNIST.

The core theoretical argument rests on the geometry of task subspaces. Forgetting is determined by the "angle" between the null spaces of different tasks. Replay alters this geometry; if it shifts the angle into a "danger zone" (approximately 45 degrees or $\pi/4$), forgetting spikes.

The "Hump" of Forgetting

A key insight is that forgetting is often non-monotonic relative to the number of replay samples. Replaying zero samples provides a baseline error. Surprisingly, replaying a few samples can cause the error to rise, creating a "hump" in the error curve. Only after a sufficient threshold of replay samples is reached does the error drop below the baseline toward zero.

MSE on Task 1 vs. Number of Replayed Examples, showing the non-monotonic 'hump' where a small amount of replay increases error for both Linear and MLP models.

This phenomenon is driven by Error Exchange. In worst-case constructions, replaying a specific sample can "transfer" error from a direction where the model was already failing to a direction it had previously mastered. Even with random sampling, if task subspaces are oriented such that the model is near a point of high sensitivity, a few constraints can pull weights into a configuration that is worse for the overall sequence.

Empirical Results: Split MNIST Failure

In experiments on Split MNIST—a class-incremental setting—the authors observed clear instances of harmful replay. In a specific sequence (distinguishing 0,6 then 1,7), replaying exactly two samples resulted in statistically significantly higher forgetting than replaying zero samples. Benefit was only observed once four or more samples were utilized.

Forgetting in Split MNIST: The blue curve (tasks 0,1 - 6,7) shows immediate benefit from replay, while the orange curve (tasks 0,6 - 1,7) shows increased forgetting when replaying 1-2 samples.

Furthermore, the effectiveness of replay is highly class-sensitive. In Rotated MNIST, replaying a '4' might assist the model, while replaying a '5' might increase forgetting. This suggests that the utility of a sample is tied to its geometric relationship with the new task.

Conclusion and Future Work

This work provides a mathematical proof (Theorem 3.2) showing that forgetting can jump from a vanishingly small value ($O(1/T)$) to a catastrophic constant value ($\Theta(1)$) simply by introducing a small amount of replay.

While the paper identifies that replay can be harmful, it does not yet offer a real-time "safety check" algorithm for practitioners to filter harmful samples. Future research may look toward cognitive science—specifically Retrieval-Induced Forgetting in humans—to develop more sophisticated strategies for memory selection in artificial agents.

Summary updated

Abstract

Continual learning seeks to enable machine learning systems to solve an increasing corpus of tasks sequentially. A critical challenge for continual learning is forgetting, where the performance on previously learned tasks decreases as new tasks are introduced. One of the commonly used techniques to mitigate forgetting, sample replay, has been shown empirically to reduce forgetting by retaining some examples from old tasks and including them in new training episodes. In this work, we provide a theoretical analysis of sample replay in an over-parameterized continual linear regression setting, where given enough replay samples, one would be able to eliminate forgetting. Our analysis focuses on replaying a few examples and highlights the role of the replay samples and task subspaces. Surprisingly, we find that forgetting can be non-monotonic with respect to the number of replay samples. We construct tasks where replay of a single example can increase forgetting and even distributions where replay of a randomly selected sample increases forgetting on average. We provide empirical evidence that this is a property of the tasks rather than the model used to train on them, by showing a similar behavior for a neural net equipped with SGD. Through experiments on a commonly used benchmark, we provide additional evidence that performance of the replay heavily depends on the choice of replay samples and the relationship between tasks.
BibTeX
@misc{mahdaviyeh2025replay,
  author = {Yasaman Mahdaviyeh and James Lucas and Mengye Ren and Andreas S. Tolias and Richard S. Zemel and Toniann Pitassi},
  title = {{Replay Can Provably Increase Forgetting}},
  year = {2025},
  eprint = {2506.04377},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.04377},
}
Figure