Selfless Sequential Learning

Rahaf Aljundi, Marcus Rohrbach, Tinne Tuytelaars

Abstract

Researchers have developed a "selfless" approach to sequential learning that prevents early tasks from "hogging" a neural network's capacity. By using a biologically inspired regularizer called SLNID, the model enforces sparsity in neuron activations, leaving plenty of "free" neurons available for future tasks and significantly reducing catastrophic forgetting.

The Challenge of Continual Learning

In the field of Continual Learning (or Lifelong Learning), the goal is to teach a single artificial intelligence model a sequence of different tasks without it forgetting the old ones—a phenomenon known as "catastrophic interference."

Most current solutions focus on protecting the parameters (weights) that are important for previous tasks. However, these methods often suffer from a "greedy" problem: the first few tasks might utilize almost all the available neurons in a fixed-capacity network. When a new task arrives, there is no "room" left. The model is then forced to either overwrite old knowledge (forgetting) or fail to learn the new task properly (underfitting). This paper asks: How can we encourage a model to be "selfless" and leave capacity open for future, unknown tasks?

Methods: The SLNID Approach

The authors introduce a novel regularization strategy called SLNID (Sparse coding through Local Neural Inhibition and Discounting). It is built on three key ideas:

  1. Neural Inhibition (SNI): Inspired by how biological brains work, this forces neurons to compete. If one neuron is highly active for a specific input, it suppresses its neighbors. This results in a sparse, decorrelated representation where only a few neurons are "on" at any given time.
  2. Locality (SLNI): Global inhibition can be too harsh for complex tasks. The authors use a Gaussian spatial weighting so that neurons only inhibit their immediate neighbors, allowing different "neighborhoods" of the network to specialize in different features.
  3. Discounting (SLNID): This is the "selfless" secret sauce. The model calculates "neuron importance" based on how sensitive the previous tasks were to specific neurons. When learning a new task, the inhibition is "discounted" for neurons already vital to old tasks. This prevents the new task from interfering with established knowledge while still encouraging the use of previously "free" neurons.

Crucially, the authors argue that enforcing sparsity in representations (which neurons fire) is far more effective for lifelong learning than enforcing sparsity in parameters (which weights are non-zero).

Parameter vs. Representation Sparsity

As illustrated above, parameter sparsity (a) may free up individual connections, but representation sparsity (b) frees up entire neurons, leaving distinct, unentangled capacity for subsequent tasks (T2).

Key Insights

  • Representation Sparsity > Parameter Sparsity: While many researchers focus on pruning weights to save space, this paper shows that having fewer active neurons is the real key to leaving capacity for future learning.
  • The "Selfless" Philosophy: Traditional training is "selfish"—it uses every resource available to minimize loss on the current task. By being "selfless," the model accepts a slightly less dense representation now to ensure it can keep learning later.
  • Biological Realism: The mammalian brain operates with 90-95% sparsity. By mimicking "lateral inhibition," artificial networks become more robust and less prone to the "entanglement" that causes forgetting.
  • Efficiency: The authors demonstrate that a smaller network using SLNID can actually outperform a network twice its size that was trained normally, because the SLNID network uses its "real estate" much more efficiently.

Experimental Results

The researchers evaluated their method on several standard and complex benchmarks for sequential learning, including Permuted MNIST, Split CIFAR-100, Split Tiny ImageNet, and an 8-Task Object Recognition Sequence.

Accuracy across sequential tasks
  • Consistent Improvements: As seen in the performance chart, SLNID (blue bars) consistently outperformed state-of-the-art methods like EWC, MAS, and other regularizers across sequential tasks.
  • CIFAR-100 Performance: On the 10-task CIFAR sequence, SLNID achieved an average accuracy improvement of 4% to 8% over non-regularized baselines.
  • State-of-the-Art: On an 8-task real-world object recognition challenge, the SLNID-enhanced model achieved the highest average accuracy (54.50%) compared to other leading lifelong learning approaches.

Limitations and Future Work

  • Fully Connected Focus: In this paper, the regularizer was primarily applied to the neurons in fully connected layers. Integrating this into convolutional layers is a logical next step.
  • Hyperparameter Tuning: The method introduces a "scale" parameter for the local inhibition, which currently requires manual tuning based on the layer size.
  • Fixed Architecture: The study assumes a fixed model capacity. Future work could explore how "selfless" learning interacts with models that are allowed to dynamically grow over time.

Summary updated

Abstract

Sequential learning, also called lifelong learning, studies the problem of learning tasks in a sequence with access restricted to only the data of the current task. In this paper we look at a scenario with fixed model capacity, and postulate that the learning process should not be selfish, i.e. it should account for future tasks to be added and thus leave enough capacity for them. To achieve Selfless Sequential Learning we study different regularization strategies and activation functions. We find that imposing sparsity at the level of the representation (i.e. neuron activations) is more beneficial for sequential learning than encouraging parameter sparsity. In particular, we propose a novel regularizer, that encourages representation sparsity by means of neural inhibition. It results in few active neurons which in turn leaves more free neurons to be utilized by upcoming tasks. As neural inhibition over an entire layer can be too drastic, especially for complex tasks requiring strong representations, our regularizer only inhibits other neurons in a local neighbourhood, inspired by lateral inhibition processes in the brain. We combine our novel regularizer with state-of-the-art lifelong learning methods that penalize changes to important previously learned parts of the network. We show that our new regularizer leads to increased sparsity which translates in consistent performance improvement on diverse datasets.
BibTeX
@inproceedings{aljundi2019selfless,
  author = {Rahaf Aljundi and Marcus Rohrbach and Tinne Tuytelaars},
  title = {{Selfless Sequential Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=imbugwwSvdq},
}
Figure