Maintaining Plasticity in Deep Continual Learning

Shibhansh Dohare, J. Fernando Hernandez-Garcia, Parash Rahman, Richard S. Sutton, A. Rupam Mahmood

TL;DR

Deep learning models, designed for single-pass training, suffer from "loss of plasticity" in continual learning scenarios, meaning they not only forget old information but also struggle to learn new tasks. This paper demonstrates this phenomenon across various datasets and architectures, and introduces "continual backpropagation," a novel algorithm that effectively maintains plasticity by selectively reinitializing underused network units.

What Problem Does This Paper Address?

Modern deep learning excels in "train-once" settings where a model is trained on a fixed dataset and then deployed. However, many real-world applications, like robotics or evolving recommendation systems, require models to learn continuously from a stream of new data. This is known as continual learning. While "catastrophic forgetting" (losing previously learned information) is a well-known challenge in continual learning, this paper focuses on a more fundamental issue: "loss of plasticity." This refers to the model's diminishing ability to learn new information, effectively becoming rigid and unable to adapt to changing data distributions. The authors aim to demonstrate the severity and generality of this loss of plasticity in current deep learning methods and propose a solution.

Methods

The paper introduces continual backpropagation (CBP), an algorithm designed to maintain plasticity. CBP builds upon standard backpropagation by incorporating a selective reinitialization mechanism. At each training step, CBP identifies "low-utility" hidden units (those contributing little to the network's output or showing little adaptation) and reinitializes a small fraction of them. This process injects fresh capacity into the network, allowing it to learn new tasks without forgetting old ones. The utility of a unit is measured by a combination of its "contribution utility" (how much it influences its consumers) and "adaptation utility" (how quickly it can change its function).

The authors also investigate existing methods for mitigating loss of plasticity, including L²-regularization, dropout, online normalization, shrink-and-perturb, and Adam. They systematically evaluate these methods on their proposed continual learning benchmarks.

Datasets

The paper utilizes the following datasets:

  • ImageNet: Repurposed into a sequence of 2000 binary classification tasks. This dataset is used to demonstrate severe loss of plasticity in a large-scale, complex setting.
  • MNIST: Repurposed into "Online Permuted MNIST" tasks. This dataset is used to test the generality of loss of plasticity across a computationally cheaper problem, allowing for more extensive hyperparameter sweeps and analysis.
  • Slowly-Changing Regression Problem: A synthetic, computationally inexpensive regression problem used to study loss of plasticity with various activation functions and to analyze the underlying causes.

Key Insights and Ideas

  • Loss of Plasticity is a Fundamental Problem: Beyond catastrophic forgetting, deep learning models fundamentally lose their ability to learn new things in continual learning settings. This is a more critical issue for long-term adaptation.
  • Train-Once Practices Hinder Continual Learning: Standard deep learning practices, optimized for single-pass training, are ill-suited for continuous adaptation.
  • Initialization is Key to Plasticity: The initial random weights of a network provide crucial properties (like diverse units and small magnitudes) that enable plasticity. These properties are lost over time with standard training.
  • Correlates of Loss of Plasticity: The paper identifies three key phenomena that accompany loss of plasticity: an increase in "dead units" (units that output zero), a growth in average weight magnitude, and a decrease in the "effective rank" of the network's representation.
  • Continual Reinitialization is Effective: The proposed "continual backpropagation" algorithm, by selectively reinitializing low-utility units, effectively combats these correlates and maintains plasticity.
  • Some Popular Methods Worsen Plasticity: Surprisingly, common techniques like Adam and dropout can exacerbate loss of plasticity, highlighting the need for specialized continual learning algorithms.

Key Results

  • ImageNet Performance Drop: In the Continual ImageNet task, binary classification accuracy dropped from 89% on an early task to 77% on the 2000th task, approaching the performance of a linear network.
  • Robustness Across Architectures: Loss of plasticity was observed across a wide range of deep network architectures, optimizers, and activation functions.
  • Shrink-and-Perturb and L²-Regularization Help: These methods were found to mitigate loss of plasticity, with shrink-and-perturb being particularly effective.
  • Continual Backpropagation Solves the Problem: CBP demonstrated sustained high performance on Continual ImageNet (maintaining accuracy above 90% over 5000 tasks) and Online Permuted MNIST, outperforming all other tested methods. It also showed minimal loss of plasticity in the Slowly-Changing Regression problem across various activations.
  • CBP Mitigates Correlates: CBP effectively maintained low weight magnitudes, a low percentage of dead units, and a high effective rank, directly addressing the identified causes of plasticity loss.

Limitations and Future Work

  • Utility Measure Heuristics: The authors acknowledge that their utility measure for identifying low-utility units is based on heuristics and could be improved with more principled approaches, potentially by propagating utility backwards from the loss function.
  • Connection Utility Adaptation: Adapting existing connection utility measures from network pruning literature for continual learning is suggested as a promising direction.
  • Dynamic Sparse Training: The connection to dynamic sparse training is noted, suggesting future work in this area for maintaining plasticity.
  • Memory Constraints: While methods that dynamically increase network size are mentioned, they are not directly comparable to CBP, which operates under finite memory constraints. Adapting these methods for finite memory is a future direction.
  • Formalization of Concepts: The need for formal definitions of phenomena like loss of plasticity and catastrophic forgetting is highlighted as crucial for future research.
  • Reinforcement Learning Complexity: While preliminary experiments in RL are presented, a full exploration of loss of plasticity in RL is acknowledged as complex due to additional confounders, requiring extensive parameter sweeps.

Summary updated

Abstract

Modern deep-learning systems are specialized to problem settings in which training occurs once and then never again, as opposed to continual-learning settings in which training occurs continually. If deep-learning systems are applied in a continual learning setting, then it is well known that they may fail to remember earlier examples. More fundamental, but less well known, is that they may also lose their ability to learn on new examples, a phenomenon called loss of plasticity. We provide direct demonstrations of loss of plasticity using the MNIST and ImageNet datasets repurposed for continual learning as sequences of tasks. In ImageNet, binary classification performance dropped from 89\% accuracy on an early task down to 77\%, about the level of a linear network, on the 2000th task. Loss of plasticity occurred with a wide range of deep network architectures, optimizers, activation functions, batch normalization, dropout, but was substantially eased by $L^2$-regularization, particularly when combined with weight perturbation. Further, we introduce a new algorithm -- continual backpropagation -- which slightly modifies conventional backpropagation to reinitialize a small fraction of less-used units after each example and appears to maintain plasticity indefinitely.
BibTeX
@misc{dohare2023maintaining,
  author = {Shibhansh Dohare and J. Fernando Hernandez-Garcia and Parash Rahman and Richard S. Sutton and A. Rupam Mahmood},
  title = {{Maintaining Plasticity in Deep Continual Learning}},
  year = {2023},
  eprint = {2306.13812},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.13812},
}
Figure