Gradient based sample selection for online continual learning

Rahaf Aljundi, Min Lin, Baptiste Goujaud, Yoshua Bengio

Overview

A continual learning agent learns online with a non-stationary and never-ending stream of data. The key to such a learning process is overcoming catastrophic forgetting—a well-known problem where neural networks overwrite previously learned information when trained on new data. To prevent this, a replay buffer is usually employed to store previous data for rehearsal.

While previous work often depends on explicit task boundaries and i.i.d. assumptions to select samples for the replay buffer, this paper introduces a principled way to populate replay buffers without relying on task boundaries. By framing sample selection as a constraint reduction problem, the authors show that keeping a diverse set of samples—specifically those with diverse parameter gradients—effectively preserves the "feasible region" of the optimization space, significantly reducing catastrophic forgetting.

The Problem: Catastrophic Forgetting in Non-Stationary Streams

Most "continual learning" (CL) solutions rely on explicit signals telling the model when one task ends and another begins. However, real-world agents encounter a continuous, non-stationary stream of data where boundaries are blurry or non-existent.

While "replay buffers" (storing a few old examples to re-train on later) are a standard fix, most methods fill these buffers using simple random or reservoir sampling. These heuristics often fail to capture the most informative samples, especially when the data stream is imbalanced or highly complex.

Methods: A Constrained Optimization View

The authors treat continual learning as a constrained optimization problem: the goal is to minimize the loss on the current sample without increasing the loss on any previous samples.

  1. Constraint Reduction: Since a model cannot store every previous sample (and thus every constraint), it must select a subset $M$ that best approximates the "feasible region"—the set of weight updates that do not hurt old tasks.
  2. Solid Angle Minimization: Geometrically, the feasible region is a polyhedral cone. To keep this region as tight as possible (to best represent the original constraints), the authors propose minimizing its solid angle.
Geometrical representation of the feasible region as a polyhedral cone.
  1. Gradient Diversity: The paper mathematically proves that minimizing this solid angle is equivalent to maximizing the diversity of gradients of the samples in the buffer.
  2. Algorithms:
    • GSS-IQP: An exact method using Integer Quadratic Programming. It is theoretically sound but computationally expensive for large buffers.
  3. GSS-Greedy: A fast, heuristic alternative. It maintains a score for each sample based on its cosine similarity to other random samples in the buffer. If a new sample is "different" enough (low similarity to existing ones), it replaces a redundant sample.

Datasets & Key Insights

The authors evaluated their approach on three standard CL benchmarks modified for the strict online setting (no task boundaries, limited training iterations): Disjoint MNIST, Permuted MNIST, and Disjoint CIFAR-10.

Key Insights: * Gradients as Features: Instead of looking at raw pixels or hidden representations to determine "diversity," the gradient is the most relevant feature because it directly describes how a sample constrains the optimization process. * Immunity to Imbalance: Unlike reservoir sampling, which fills the buffer based on the frequency of data, the Gradient-based Sample Selection (GSS) approach focuses on the information or constraint provided by the data. This makes it much more robust when some tasks have many more samples than others.

Key Results

The empirical results demonstrate the clear advantage of gradient-based selection over random baselines.

Performance comparison of sample selection methods across different buffer sizes.
  • Superior Accuracy: On Disjoint MNIST with a buffer of 500 samples, GSS-Greedy achieved ~84.8% accuracy, compared to only ~57.9% for random sampling.
  • Beating Task-Aware Methods: Surprisingly, GSS-Greedy often performed on par with or better than state-of-the-art methods like GEM and iCaRL, even though those methods were given the "unfair advantage" of knowing exactly when tasks changed.
  • Robustness to Imbalance: In experiments where one task had 10x more data than others, GSS-Greedy maintained high accuracy (~78%) while reservoir sampling plummeted (~69%) because it allowed the majority task to "crowd out" the buffer.

Limitations and Future Work

  • Computational Cost: Even the "greedy" version requires computing gradients for candidate samples. While more efficient than IQP, it is still more expensive than simple random sampling.
  • Gradient Quality: The method relies on the quality of the gradients. In the very early stages of training (when the model is essentially random), the gradients might be noisy, potentially leading to sub-optimal initial sample selection.
  • Scalability: While tested on ResNet18 and CIFAR-10, the authors note that for much larger models or more complex datasets, the overhead of gradient-based scoring remains a challenge to be addressed.

Summary updated

Abstract

A continual learning agent learns online with a non-stationary and never-ending stream of data. The key to such learning process is to overcome the catastrophic forgetting of previously seen data, which is a well known problem of neural networks. To prevent forgetting, a replay buffer is usually employed to store the previous data for the purpose of rehearsal. Previous work often depend on task boundary and i.i.d. assumptions to properly select samples for the replay buffer. In this work, we formulate sample selection as a constraint reduction problem based on the constrained optimization view of continual learning. The goal is to select a fixed subset of constraints that best approximate the feasible region defined by the original constraints. We show that it is equivalent to maximizing the diversity of samples in the replay buffer with parameter gradient as the feature. We further develop a greedy alternative that is cheap and efficient. The advantage of the proposed method is demonstrated by comparing to other alternatives under the continual learning setting. Further comparisons are made against state of the art methods that rely on task boundaries which show comparable or even better results for our method.
BibTeX
@inproceedings{aljundi2019gradient,
  author = {Rahaf Aljundi and Min Lin and Baptiste Goujaud and Yoshua Bengio},
  title = {{Gradient based sample selection for online continual learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=SSeH5zyLrxq},
}
Figure