Continual Backprop: Stochastic Gradient Descent with Persistent Randomness

Shibhansh Dohare, Richard S. Sutton, A. Rupam Mahmood

TL;DR

Neural networks trained with standard Backpropagation (BP) degrade in performance over time when faced with continuously changing data distributions. This paper introduces "Continual Backprop" (CBP), an algorithm that injects random features during training to maintain adaptability, enabling robust performance in non-stationary environments.

What Problem Does This Paper Address?

Deep learning models, powered by Backpropagation (BP), have achieved state-of-the-art results on many tasks. However, BP's effectiveness relies heavily on initial random weight initialization. This paper highlights a critical limitation: when applied to "continual learning" scenarios – where the data distribution changes over time (non-stationary problems) – BP's performance degrades significantly. This degradation stems from the fact that the initial randomness, crucial for learning, is not maintained throughout the learning process. Standard BP, therefore, struggles to adapt to evolving environments, making it unsuitable for many real-world applications like robotics or dynamic systems. The paper aims to understand this "decaying plasticity" and propose a solution that allows neural networks to continually adapt.

Methods

The core contribution is the Continual Backprop (CBP) algorithm. CBP builds upon standard Backpropagation by incorporating a generate-and-test process that continually injects random features into the network. This process has two main components:

  • Generator: Proposes new random features, sampled from the initial weight distribution.
  • Tester: Identifies low-utility features within the network and replaces them with newly generated ones.

This mechanism aims to preserve the benefits of initial random initialization throughout the entire learning process, ensuring the network remains plastic and adaptable. The paper also introduces a novel utility measure for features, combining their "contribution utility" (how much they influence subsequent features) and "adaptation utility" (how quickly they can adapt based on incoming weights).

Datasets

The paper evaluates its methods on the following datasets/benchmarks:

  • Bit-Flipping Problem: A novel, idealized semi-stationary problem designed to study continual learning algorithms efficiently. It involves a non-stationary input distribution while the target function remains stationary.
  • Permuted MNIST: An online image classification task where images are presented sequentially, and the permutation of pixels changes after each epoch, creating a long-term continual learning problem.
  • Slippery Ant: A non-stationary reinforcement learning (RL) problem, a variant of the PyBullet Ant environment, where the friction between the agent and the ground changes periodically.

Key Insights and Ideas

  • Backprop's "Decaying Plasticity": The paper provides the first empirical evidence that standard Backprop's ability to learn degrades over time in non-stationary environments, even with careful initialization. This is a fundamental limitation for continual learning.
  • Initial Randomness is Key, But Not Persistent: The initial random weights are crucial for BP's initial success, but their benefits are lost as learning progresses without a mechanism to reintroduce randomness.
  • Continual Feature Injection as a Solution: CBP's generate-and-test approach, by continually injecting random features, effectively extends the benefits of initial randomness, allowing for sustained adaptation.
  • Utility Measure is Crucial: The proposed utility measure, combining contribution and adaptation, is essential for effectively identifying and replacing low-utility features in CBP.
  • Generalizability: CBP demonstrates effectiveness across supervised learning (Bit-Flipping, Permuted MNIST) and reinforcement learning (Slippery Ant) problems, suggesting its broad applicability.

Key Results

  • Degradation in Backprop: In the Bit-Flipping problem, Backprop's error rate increases significantly over time for various activation functions and optimizers (SGD and Adam), even with smaller step-sizes.
  • CBP's Robustness: Continual Backprop (CBP) consistently maintains a low error rate and high accuracy across all tested non-stationary problems, performing comparably to its initial performance.
  • Permuted MNIST Performance: CBP maintains good online classification accuracy on Permuted MNIST, while other methods like BP, BP+L2, and BP+OnlineNorm show performance degradation.
  • Slippery Ant Performance: In the non-stationary RL setting, Continual PPO (CPPO), which uses CBP principles, outperforms standard PPO and PPO+L2, showing sustained high episodic returns.
  • Parameter Sensitivity: CBP is shown to be less sensitive to its hyperparameters (replacement-rate) compared to BP+L2's weight-decay.

Limitations and Future Work

  • Heuristic-Based Tester: The paper acknowledges that the tester component of CBP's generate-and-test process is based on heuristics. Future work could focus on developing more principled testers to improve the algorithm's foundations.
  • Scalability of Compute: The experiments, particularly for the Bit-Flipping problem, required significant computational resources (thousands of CPU-days). While CBP is efficient in terms of memory, further optimization for computational speed might be beneficial.
  • Understanding Feature Saturation: While feature saturation and vanishing gradients are observed, they don't fully explain the performance differences across all algorithms, suggesting that other factors are at play. Further research is needed to fully understand these phenomena.

Summary updated

Abstract

The Backprop algorithm for learning in neural networks utilizes two mechanisms: first, stochastic gradient descent and second, initialization with small random weights, where the latter is essential to the effectiveness of the former. We show that in continual learning setups, Backprop performs well initially, but over time its performance degrades. Stochastic gradient descent alone is insufficient to learn continually; the initial randomness enables only initial learning but not continual learning. To the best of our knowledge, ours is the first result showing this degradation in Backprop’s ability to learn. To address this issue, we propose an algorithm that continually injects random features alongside gradient descent using a new generate-and-test process. We call this the Continual Backprop algorithm. We show that, unlike Backprop, Continual Backprop is able to continually adapt in both supervised and reinforcement learning problems. We expect that as continual learning becomes more common in future applications, a method like Continual Backprop will be essential where the advantages of random initialization are present throughout learning.
BibTeX
@inproceedings{dohare2022continual,
  author = {Shibhansh Dohare and Richard S. Sutton and A. Rupam Mahmood},
  title = {{Continual Backprop: Stochastic Gradient Descent with Persistent Randomness}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=86sEVRfeGYS},
}
Figure