Replay can provably increase forgetting

Yasaman Mahdaviyeh, James Lucas, Mengye Ren, Andreas S. Tolias, Richard Zemel, Toniann PitassiSubmission to ICLR 2025 · Withdrawn

TL;DR

Experience replay—storing and re-training on a small buffer of past examples—is one of the most widely used strategies to combat catastrophic forgetting in continual learning. This paper proves mathematically and demonstrates empirically that replaying a small number of samples can counter-intuitively increase forgetting compared to no replay at all. Forgetting can be non-monotonic with respect to the replay buffer size: adding a few replay samples can make performance worse before adding many samples finally makes it better.

What Problem Does This Paper Address?

In continual learning, models learn tasks sequentially and frequently suffer from catastrophic forgetting, where acquiring new capabilities erases previously acquired knowledge. Storing a small episodic memory buffer of past examples and mixing them into future training batches (sample replay) is an intuitive and standard remedy.

While replaying enough samples eventually eliminates forgetting, real-world systems operate under strict memory constraints and can only retain tiny subsets of data. Previous theoretical work established that cyclic full-task replay eliminates forgetting, but the small-buffer regime has remained largely unexamined theoretically. The authors ask: Can replaying a tiny number of past samples actually backfire and worsen forgetting?

Methods

The authors investigate over-parameterized continual linear regression solved with gradient descent/SGD (which implicitly finds minimum-norm solutions to $X_t w = y_t$):

  • Iterate Geometry: Parameter error updates via recursive projections onto the null spaces of sequential tasks: $(w_t - w^*) = P_t (w_{t-1} - w^*)$. Forgetting on past tasks depends directly on how the error vector aligns with the task subspaces.
  • Worst-Case Adversarial Analysis: The authors construct a minimal 3D geometric setup ($T$ tasks in $\mathbb{R}^d$) with specific task subspaces and select an adversarial replay sample.
  • Average-Case Random Setting: The authors analyze a stochastic setting where task data is drawn from Gaussian distributions supported on low-rank subspaces, and replay samples are drawn uniformly at random.
  • Empirical Extension to Deep Learning: They test whether this geometric phenomenon persists in non-linear models by training Multi-Layer Perceptrons (MLPs with ReLU activations) using Adam on synthetic linear regression tasks and classification benchmarks.

Datasets

  • Synthetic Linear Regression Tasks:
  • Low-dimensional ($d=3$) and higher-dimensional ($d=10, 50$) synthetic tasks with controlled subspace angles and Gaussian-distributed samples.
  • Rotated MNIST: Sequential tasks where MNIST digits are rotated (45° and 90° rotations) in a task-incremental setting.
  • Split MNIST: Binary classification splits in a class-incremental setting (comparing sequences such as $\{0 \text{ vs. } 1\} \to \{6 \text{ vs. } 7\}$ against $\{0 \text{ vs. } 6\} \to \{1 \text{ vs. } 7\}$).

Key Insights and Ideas

  • Error Exchange Mechanism: Replaying an example alters the optimization update trajectory. While training on a replayed sample eliminates error along that specific sample's direction, the update can rotate parameter error into the span of other, un-replayed samples from past tasks. In the worst-case construction, replaying a single sample converts vanishing $O(1/T)$ forgetting into catastrophic $\Theta(1)$ forgetting.
  • Average-Case Deterioration: The degradation is not just a fragile adversarial fluke. In a 2-task setup where samples and replay choices are randomly drawn from task subspaces, expected forgetting with replay is provably higher than without replay ($\mathbb{E}[F(w_2)] < \mathbb{E}[F(\tilde{w}_2)]$).
  • Null-Space Angle Non-monotonicity: Forgetting depends non-monotonically on the angle between task null spaces. Adding a replay sample shrinks the dimension of the second task's null space but shifts its orientation closer to the angle that maximizes projection error onto the first task.
  • Replay Sensitivity to Task Relationships: The efficacy of replaying isolated examples depends heavily on which class is replayed and the structural similarity between tasks.

Key Results

  • Theoretical Bounds: The authors prove analytically that replaying one sample can jump forgetting from $O(1/T)$ to $\Theta(1)$ in the worst case, and that random replay in an average-case Gaussian task setting yields an expected forgetting ratio $> 1.4\times$ over no replay.
  • Non-Linear MLPs on Synthetic Data: In 3D and 50D linear tasks, both linear models and 1-hidden-layer MLPs exhibit an initial spike in Mean Squared Error (MSE) on Task 1 when replaying 1–10 samples, only dropping below the no-replay baseline after buffer size increases substantially (Figure 2).
  • Split MNIST: When learning $\{0, 6\}$ followed by $\{1, 7\}$, replaying 1 or 2 samples increases forgetting compared to zero replay (statistically significant with Welch’s $t$-test, $p < 0.05$ over 80 runs), only providing a net benefit once 4 or more samples are replayed.
  • Rotated MNIST: Replay efficacy varies drastically by class. At 45° rotation, replaying a single '4' reduces forgetting by ~3%, while replaying a '5' often increases forgetting relative to fine-tuning with no replay (Figures 3 & 4).

Limitations and Future Work

  • Constrained Settings: The closed-form theoretical proofs focus on 2-task or specific multi-task sequences in over-parameterized linear regression, though empirical results demonstrate similar trends in small neural networks.
  • Characterization Bounds: The paper establishes existence proofs (that replay can increase forgetting), but does not fully formulate necessary and sufficient conditions for when this occurs in general architectures.
  • Buffer Selection Algorithms: The study primarily looks at random selection and worst-case selection of tiny buffers without proposing an active sample-selection strategy that guarantees monotonic forgetting reduction when future task geometries are unknown.

Summary updated

Abstract

Continual learning seeks to enable machine learning systems to solve an increasing corpus of tasks sequentially. A critical challenge for continual learning is forgetting, where the performance on previously learned tasks decreases as new tasks are introduced. One of the commonly used techniques to mitigate forgetting, sample replay, has been shown empirically to reduce forgetting by retaining some examples from old tasks and including them in new training episodes. In this work, we provide a theoretical analysis of sample replay in an over-parameterized continual linear regression setting, where given enough replay samples, one would be able to eliminate forgetting. Our analysis focuses on replaying a few examples and highlights the role of the replay samples and task subspaces. Surprisingly, we find that forgetting can be non-monotonic with respect to the number of replay samples. We construct tasks where replay of a single example can increase forgetting and even distributions where replay of a randomly selected sample increases forgetting on average. We provide empirical evidence that this is a property of the tasks rather than the model used to train on them, by showing a similar behavior for a neural net equipped with SGD. Through experiments on a commonly used benchmark, we provide additional evidence that performance of the replay heavily depends on the choice of replay samples and the relationship between tasks.
BibTeX
@misc{mahdaviyeh2025replay,
  author = {Yasaman Mahdaviyeh and James Lucas and Mengye Ren and Andreas S. Tolias and Richard Zemel and Toniann Pitassi},
  title = {{Replay can provably increase forgetting}},
  year = {2025},
  eprint = {2506.04377},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.04377},
}
Figure