Elephant Neural Networks: Born to Be a Continual Learner

Qingfeng Lan, A. Rupam Mahmood

Abstract

Catastrophic forgetting remains a significant challenge to continual learning for decades. While recent works have proposed effective methods to mitigate this problem, they mainly focus on the algorithmic side. Meanwhile, we do not fully understand what architectural properties of neural networks lead to catastrophic forgetting. This study aims to fill this gap by studying the role of activation functions in the training dynamics of neural networks and their impact on catastrophic forgetting. Our study reveals that, besides sparse representations, the gradient sparsity of activation functions also plays an important role in reducing forgetting. Based on this insight, we propose a new class of activation functions, elephant activation functions, that can generate both sparse representations and sparse gradients. We show that by simply replacing classical activation functions with elephant activation functions, we can significantly improve the resilience of neural networks to catastrophic forgetting. Our method has broad applicability and benefits for continual learning in regression, class incremental learning, and reinforcement learning tasks. Specifically, we achieve excellent performance on the Split MNIST dataset in just one single pass, without using replay buffers, task boundary information, or pre-training.

Introduction

The "Achilles' heel" of artificial neural networks is catastrophic forgetting. When a standard model learns a new task, it often completely overwrites the weights used for its previous tasks. This occurs because standard training algorithms induce global changes; an update intended to correct a mistake on a new data point propagates through the entire architecture, corrupting existing knowledge.

Traditional solutions often rely on "algorithmic band-aids," such as replay buffers (storing and re-training on old data) or task boundaries (explicitly signaling task shifts). However, these methods are memory-intensive and often impractical for real-world "streaming" scenarios. This research investigates whether a network architecture can be designed to be inherently resistant to forgetting by focusing on the properties of activation functions.

Methods: Achieving Local Elasticity

A key requirement for a robust continual learner is local elasticity. This property ensures that an update at one input point primarily affects the network's output in a small neighborhood around that point, leaving distant memories intact.

Through mathematical analysis using the Neural Tangent Kernel (NTK), the authors demonstrate that sparse representations (where most neurons are inactive) are insufficient for non-linear networks to avoid forgetting. Crucially, the network also requires sparse gradients. In standard architectures using ReLU or Sigmoid, the gradient remains non-zero over wide ranges, causing "forgetting" signals to propagate globally.

To address this, the authors introduce the Elephant activation function:

$$Elephant(x) = \\frac{1}{1 + |x/a|^d}$$

The Elephant activation function and its gradient. (a) Visual metaphor of an elephant inside a boa constrictor. (b) The bell-shaped activation function. (c) The resulting sparse gradient.

Unlike ReLU, which has a constant gradient for half its range, the Elephant function is bell-shaped. It is only active within a narrow window (controlled by parameter $a$), and its gradient drops to zero rapidly outside that window (controlled by $d$). This creates a "surgical" update mechanism, allowing the model to edit its knowledge point-by-point without global interference.

Experimental Evaluation

The efficacy of Elephant Neural Networks (ENNs) was evaluated across three distinct domains:

  • Streaming Regression: In approximating a sine wave from sequential data, ENNs achieved a Mean Squared Error (MSE) of 0.0081, whereas standard ReLU and Sigmoid models failed completely (MSE > 0.45).
  • Class Incremental Learning: On the Split MNIST benchmark, ENNs achieved ~85% accuracy in a single pass without replay buffers or task boundary information, outperforming standard CNNs (~77%) and specialized algorithms like Streaming EWC (~78%).
  • Reinforcement Learning (RL): In classic control tasks (e.g., MountainCar, Acrobot), an ENN using a minimal replay buffer of only 32 samples matched the performance of a standard MLP requiring 10,000 samples—a 300x reduction in memory requirements.

Conclusion and Future Work

While ENNs offer a powerful architectural solution to forgetting, several challenges remain. The parameters $a$ (width) and $d$ (slope) are sensitive to the input data distribution and currently require manual tuning. Furthermore, the authors found that replacing activations in the last hidden layer was most effective; applying the function to all layers can lead to a "loss of plasticity," making it difficult for the network to learn complex features from scratch. Future work will explore scaling these functions to larger architectures like Transformers and broader datasets like ImageNet.

Summary updated

Abstract

Catastrophic forgetting remains a significant challenge to continual learning for decades. While recent works have proposed effective methods to mitigate this problem, they mainly focus on the algorithmic side. Meanwhile, we do not fully understand what architectural properties of neural networks lead to catastrophic forgetting. This study aims to fill this gap by studying the role of activation functions in the training dynamics of neural networks and their impact on catastrophic forgetting. Our study reveals that, besides sparse representations, the gradient sparsity of activation functions also plays an important role in reducing forgetting. Based on this insight, we propose a new class of activation functions, elephant activation functions, that can generate both sparse representations and sparse gradients. We show that by simply replacing classical activation functions with elephant activation functions, we can significantly improve the resilience of neural networks to catastrophic forgetting. Our method has broad applicability and benefits for continual learning in regression, class incremental learning, and reinforcement learning tasks. Specifically, we achieves excellent performance on Split MNIST dataset in just one single pass, without using replay buffer, task boundary information, or pre-training.
BibTeX
@misc{lan2023elephant,
  author = {Qingfeng Lan and A. Rupam Mahmood},
  title = {{Elephant Neural Networks: Born to Be a Continual Learner}},
  year = {2023},
  eprint = {2310.01365},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.01365},
}
Figure