Abstract
Experience replay is widely considered the "gold standard" for mitigating catastrophic forgetting in continual learning. However, this research demonstrates that replay can be a double-edged sword. Through theoretical analysis of over-parameterized linear regression and empirical validation with neural networks, we show that replaying a small number of samples can paradoxically increase forgetting compared to no replay at all.
Introduction
When machine learning models learn tasks sequentially, they often suffer from catastrophic forgetting: mastering a new task (Task B) leads to the loss of performance on a previous one (Task A). The standard remedy is experience replay, where a buffer of old data is mixed into current training. While generally effective, the theoretical foundations of replay are limited. This paper challenges the assumption that "some replay is always better than none," identifying specific scenarios where replay is counterproductive.
Methodology
The study approaches the problem through two distinct lenses:
- Theoretical Analysis: The authors examine over-parameterized linear regression. While simple enough for closed-form mathematical solutions via orthogonal projections, this setting is complex enough to exhibit catastrophic forgetting. Analysis covers both worst-case task sequences and average-case scenarios using random task subspaces.
- Empirical Validation: To ensure findings extend beyond linear models, Multi-Layer Perceptrons (MLPs) were tested using standard optimizers (SGD and Adam) on classic benchmarks like Rotated MNIST and Split MNIST.
The core theoretical argument rests on the geometry of task subspaces. Forgetting is determined by the "angle" between the null spaces of different tasks. Replay alters this geometry; if it shifts the angle into a "danger zone" (approximately 45 degrees or $\pi/4$), forgetting spikes.
The "Hump" of Forgetting
A key insight is that forgetting is often non-monotonic relative to the number of replay samples. Replaying zero samples provides a baseline error. Surprisingly, replaying a few samples can cause the error to rise, creating a "hump" in the error curve. Only after a sufficient threshold of replay samples is reached does the error drop below the baseline toward zero.
This phenomenon is driven by Error Exchange. In worst-case constructions, replaying a specific sample can "transfer" error from a direction where the model was already failing to a direction it had previously mastered. Even with random sampling, if task subspaces are oriented such that the model is near a point of high sensitivity, a few constraints can pull weights into a configuration that is worse for the overall sequence.
Empirical Results: Split MNIST Failure
In experiments on Split MNIST—a class-incremental setting—the authors observed clear instances of harmful replay. In a specific sequence (distinguishing 0,6 then 1,7), replaying exactly two samples resulted in statistically significantly higher forgetting than replaying zero samples. Benefit was only observed once four or more samples were utilized.
Furthermore, the effectiveness of replay is highly class-sensitive. In Rotated MNIST, replaying a '4' might assist the model, while replaying a '5' might increase forgetting. This suggests that the utility of a sample is tied to its geometric relationship with the new task.
Conclusion and Future Work
This work provides a mathematical proof (Theorem 3.2) showing that forgetting can jump from a vanishingly small value ($O(1/T)$) to a catastrophic constant value ($\Theta(1)$) simply by introducing a small amount of replay.
While the paper identifies that replay can be harmful, it does not yet offer a real-time "safety check" algorithm for practitioners to filter harmful samples. Future research may look toward cognitive science—specifically Retrieval-Induced Forgetting in humans—to develop more sophisticated strategies for memory selection in artificial agents.
Summary updated