Abstract
Catastrophic forgetting poses a significant challenge in continual learning, where models often forget previous tasks when trained on new data. Our empirical analysis reveals a strong correlation between catastrophic forgetting and the learning speed of examples: examples learned early are rarely forgotten, while those learned later are more susceptible to forgetting. We demonstrate that replay-based continual learning methods can leverage this phenomenon by focusing on mid-learned examples for rehearsal. We introduce Goldilocks, a novel replay buffer sampling method that filters out examples learned too quickly or too slowly, keeping those learned at an intermediate speed. Goldilocks improves existing continual learning algorithms, leading to state-of-the-art performance across several image classification tasks.
1. Introduction
In the field of Continual Learning (CL), neural networks are tasked with learning a sequence of distinct tasks. A primary obstacle in this paradigm is catastrophic forgetting, where the acquisition of new knowledge (e.g., Task B) leads to the abrupt loss of previously learned information (e.g., Task A). While "replay" or "rehearsal" strategies—which maintain a small buffer of past examples—are common solutions, they often rely on random sampling. This paper investigates whether certain examples are inherently more prone to forgetting and how identifying these examples can optimize the replay process.
2. Methodology: The LIFO Pattern and Learning Speed
The researchers propose a behavioral metric called "learning speed," defined by how early and consistently a model correctly classifies an example during its initial training phase. Their empirical findings reveal a Last-In-First-Out (LIFO) forgetting pattern: examples that are learned later in the training process (slow-learned) are the first to be forgotten when the model transitions to new tasks. Conversely, examples learned quickly (fast-learned) form a "core memory" that is highly resistant to forgetting.
This distinction is often driven by a "simplicity bias," where models prioritize easy patterns. For instance, a clear, centered image of a bicycle is learned rapidly, whereas a complex scene containing a person on a bike may take significantly longer to master.
3. Speed-Based Sampling (SBS) and the Goldilocks Zone
Building on the LIFO discovery, the authors introduced Speed-Based Sampling (SBS), also known as the Goldilocks method. Instead of random selection, SBS ranks examples by their learning speed and filters out the extremes:
- Fastest-learned examples: These are "too easy" and the model is unlikely to forget them even without replay.
- Slowest-learned examples: These are often "too hard" or noisy, consuming valuable buffer space without providing robust generalizable knowledge.
The remaining "Goldilocks" examples—those learned at an intermediate speed—are prioritized for the replay buffer. To determine the optimal filtering thresholds without looking at test data, the authors utilize an auxiliary self-supervised task (RotNet) to estimate dataset-specific complexity.
4. Experimental Results and Insights
The SBS method was evaluated on standard benchmarks including CIFAR-10, CIFAR-100, and TinyImageNet across both Task-Incremental (TIL) and Class-Incremental (CIL) settings.
Key Findings:
- Performance Gains: SBS consistently outperformed random sampling and specialized methods like Herding or GSS. On a 20-task CIFAR-100 split, SBS improved final accuracy by over 3.5% compared to random baselines.
- Universality: The LIFO pattern and the effectiveness of SBS were observed across various architectures (ResNet, VGG) and optimization techniques.
- Task Independence: Interestingly, the "best" examples to retain from a task are largely independent of the nature of future tasks. Forgetting is primarily a function of the example's own properties relative to the model's learning dynamics.
5. Conclusion and Future Work
The discovery of the LIFO forgetting pattern provides a simple yet powerful heuristic for managing memory in continual learning. By focusing on the "Goldilocks" zone of learning speed, SBS offers a computationally efficient way to mitigate catastrophic forgetting.
However, a current limitation is the dependency on multi-epoch training to calculate learning speed, which poses challenges for strictly online (single-pass) continual learning. Future research aims to develop proxies for "memorable-ness" that can be computed in a single pass, further broadening the applicability of speed-based strategies.
Summary updated