Abstract
A continual learning agent learns online with a non-stationary and never-ending stream of data. The key to such learning process is to overcome the catastrophic forgetting of previously seen data, which is a well known problem of neural networks. To prevent forgetting, a replay buffer is usually employed to store the previous data for the purpose of rehearsal. Previous works often depend on task boundary and i.i.d. assumptions to properly select samples for the replay buffer. In this work, we formulate sample selection as a constraint reduction problem based on the constrained optimization view of continual learning. The goal is to select a fixed subset of constraints that best approximate the feasible region defined by the original constraints. We show that it is equivalent to maximizing the diversity of samples in the replay buffer with parameters gradient as the feature. We further develop a greedy alternative that is cheap and efficient. The advantage of the proposed method is demonstrated by comparing to other alternatives under the continual learning setting. Further comparisons are made against state of the art methods that rely on task boundaries which show comparable or even better results for our method.
Introduction: Moving Beyond Task Boundaries
Neural networks are notoriously prone to catastrophic forgetting, where learning new information overwrites the weights necessary for previous tasks. The standard solution is a "replay buffer"—a small memory of past examples revisited during training. However, most replay strategies rely on two unrealistic assumptions: that the system knows exactly when one task ends and another begins, and that data within a task is independent and identically distributed (i.i.d.).
In real-world scenarios, data streams are often blurry, non-stationary, and imbalanced. For instance, a model might see thousands of "cat" images before encountering a handful of "dogs." Standard methods like reservoir sampling fail here because they fill the buffer based on frequency, causing rare but important classes to be forgotten almost immediately. This paper proposes a gradient-based approach to sample selection that functions effectively without needing explicit task boundaries.
Methodology: Sample Selection as Constraint Reduction
The authors reframe continual learning as a constrained optimization problem. The objective is to minimize the loss on the current sample without increasing the loss on any previously seen samples. Geometrically, each past sample acts as a constraint on the model's parameter space, defining a "feasible region"—a polyhedral cone where the model can update its weights without forgetting.
Since storing every sample is impossible, the goal is to select a fixed-size subset of constraints that best approximates this feasible region. The authors prove that the optimal subset is the one that minimizes the solid angle of the cone formed by the gradients. As calculating solid angles in high dimensions is computationally prohibitive, they propose a surrogate: maximizing the diversity of gradients in the replay buffer.
Proposed Algorithms
- GSS-IQP: An exact version using Integer Quadratic Programming to select the most diverse samples. While accurate, it is computationally expensive for large buffers.
- GSS-Greedy: A fast, heuristic alternative designed for online streams. For every new sample, the algorithm compares its gradient to a random subset of the buffer. If the new sample's gradient is sufficiently "different" (low cosine similarity), it replaces a redundant sample in the buffer.
Empirical Results
The proposed Gradient-Based Sample Selection (GSS) methods were evaluated on benchmarks like Disjoint MNIST, Permuted MNIST, and CIFAR-10, modified for a task-boundary-free setting.
As shown in the results, GSS-Greedy significantly outperforms random reservoir sampling, particularly when the buffer size is small. On Disjoint MNIST with a 300-slot buffer, GSS-Greedy achieved 82.6% accuracy, compared to 37.5% for random sampling. Furthermore, GSS-Greedy performed on par with state-of-the-art methods like GEM and iCaRL, despite those methods having the advantage of knowing task boundaries.
Key Insights and Conclusion
A central insight of this work is that gradients are more informative than raw pixels for sample selection. While many methods prioritize visual diversity, gradients capture the actual constraints imposed on the optimization process. By picking gradients that point in diverse directions, the algorithm "pinches" the feasible region, forcing the model to find solutions that satisfy a wide variety of past experiences.
This approach is particularly robust to imbalanced data streams. Unlike frequency-based sampling, GSS-Greedy selects samples based on their informativeness. If a rare class appears, its gradient will likely be distinct from the "flooded" majority class, ensuring it is preserved in the buffer. While the computational cost of calculating gradients for every sample is higher than random sampling, the gains in stability and performance make it a compelling choice for real-world online continual learning.
Summary updated