Elephant Neural Networks: Born to Be a Continual Learner

Qingfeng Lan, A. Rupam MahmoodSubmission to ICLR 2024 · Rejected

TL;DR

Most continual learning methods rely on complex algorithmic workarounds like replay buffers or task boundaries to mitigate catastrophic forgetting. This paper reveals that network architecture—specifically activation functions—plays a massive role in forgetting, showing that neural networks need both sparse representations and sparse gradients. To achieve this, the authors introduce Elephant Activation Functions, bell-shaped activations that grant deep networks local elasticity, enabling effective continual learning across regression, classification, and reinforcement learning with minimal or no replay buffers.

What Problem Does This Paper Address?

When artificial neural networks learn sequentially from non-stationary (non-iid) data, they suffer from catastrophic forgetting: updating parameters on new data erases previously acquired knowledge. While existing research has focused on external algorithmic fixes (such as storing past examples in replay buffers, regularizing important weights like EWC, or dynamically isolating sub-networks), the fundamental question of why standard architectures forget so aggressively remains under-explored.

The authors investigate the training dynamics of neural networks through the lens of the Neural Tangent Kernel (NTK) and local elasticity—the ability of a network to update its prediction on a new input without altering predictions on dissimilar inputs.

Methods

Analyzing the NTK of a single-hidden-layer network $f_{\mathbf{w}}(\mathbf{x}) = \mathbf{u}^\top \sigma(\mathbf{V}\mathbf{x} + \mathbf{b})$, the parameter update under gradient descent reveals two interaction terms: $$\langle \nabla_{\mathbf{w}} f(\mathbf{x}), \nabla_{\mathbf{w}} f(\mathbf{x}_t) \rangle = \sigma(\mathbf{V}\mathbf{x} + \mathbf{b})^\top \sigma(\mathbf{V}\mathbf{x}_t + \mathbf{b}) + \mathbf{u}^\top\mathbf{u}(\mathbf{x}^\top \mathbf{x}_t + 1)\sigma'(\mathbf{V}\mathbf{x} + \mathbf{b})^\top \sigma'(\mathbf{V}\mathbf{x}_t + \mathbf{b})$$

  • The Failure of Sparse Representations Alone: Prior works often rely solely on sparse feature activations ($\sigma(\cdot) \approx \mathbf{0}$). However, the second term involves the gradients of the activation function ($\sigma'(\cdot)$). If the derivative $\sigma'$ is dense, updating on $\mathbf{x}_t$ still causes non-local weight changes, breaking local elasticity.
  • Elephant Activation Function: The authors introduce a bell-shaped activation function designed to enforce sparsity in both function values and derivative values: $$\text{Elephant}(x) = \frac{1}{1 + \left|\frac{x}{a}\right|^d}$$
  • $a$ modulates the function width (controlling representation sparsity).
  • $d$ controls the steepness of the slope (controlling gradient sparsity).
  • Elephant Neural Networks (ENNs): In practice, standard activations (like ReLU) in the final hidden layer are replaced with Elephant functions (e.g., EMLP, ECNN). Bias terms in this layer are initialized evenly across an interval to ensure feature diversity.

Datasets

The authors evaluate ENNs across three distinct continual learning paradigms:

  • Streaming 1D Regression: Continuous streaming approximation of a sine wave $y = \sin(\pi x)$ over $x \in [0, 2]$ in a single pass ($n = 200$ points, evaluated on 1,000 evenly spaced test points).
  • Class-Incremental Learning: Evaluated under a strict streaming setting (single pass, no replay buffer, no task boundaries, no pre-training):
  • Split MNIST (5 tasks of 2 classes each).
  • Split CIFAR-10 (5 tasks of 2 classes each).
  • Pre-trained embedding variants: Split Embedding CIFAR-10, Split Embedding CIFAR-100, and Split Embedding Tiny ImageNet (using frozen ConvMixer representations).
  • Reinforcement Learning: Four continuous-state, discrete-action control benchmarks from OpenAI Gym and PyGame Learning Environment under extreme buffer constraints: MountainCar-v0, Acrobot-v1, Catcher, and Pixelcopter.

Key Insights and Ideas

  • Sparse representations need sparse gradients: In linear models, representation sparsity suffices for local updates. In non-linear deep networks, gradients of the feature extractor propagate across the network unless $\sigma'(\cdot)$ is also sparse.
  • True point-wise editing: Standard MLPs exhibit global generalization—updating a prediction at $x = 1.5$ drastically shifts predictions across the entire input domain. ENNs exhibit strong local elasticity, altering predictions exclusively in a narrow window around the updated point.
  • Architectural fixes outperform algorithmic patches: A standard network augmented with Streaming EWC yields only modest gains, whereas switching the architecture to an Elephant network provides substantially higher resilience to forgetting in single-pass settings.

Key Results

  • Streaming Regression: On the streaming sine task, classical MLPs (ReLU, Sigmoid, Tanh, ELU) fail with test MSEs ranging from $0.446$ to $0.473$. Sparse-representation networks (SR-NN) achieve $0.406$. EMLP achieves an MSE of $0.0081 \pm 0.0009$.
  • Class-Incremental Learning (Split MNIST):
  • With a 10K-neuron hidden layer in a single pass without task boundaries: standard MLP reaches $62.1\%$ and MLP + Streaming EWC reaches $60.9\%$.
  • EMLP achieves $80.2 \pm 0.2\%$, and ECNN achieves $85.0 \pm 0.4\%$, approaching FlyModel ($91\%$), which relies on explicit task boundary awareness.
  • On Split CIFAR-10 (10K neurons), ECNN achieves $24.3\%$ vs. CNN's $15.1\%$ and CNN + Streaming EWC's $17.9\%$.
  • Reinforcement Learning: In DQN agent training, an EMLP using a tiny replay buffer ($m = 32$) outperforms standard MLPs with the same buffer size across all 4 environments and matches or rivals standard MLPs using a massive replay buffer ($m = 10,000$) on MountainCar, Acrobot, and Catcher.

Limitations and Future Work

  • Hyperparameter Sensitivity: There is currently no analytical method to set the function width $a$ or the bias initialization spread $\sigma_{\text{bias}}$; they must be tuned empirically based on the input dimension and data distribution.
  • Loss of Plasticity in Deep Layers: Initial experiments showed that replacing activations across all layers hurt performance, likely due to optimization difficulties and loss of plasticity. Consequently, Elephant activations were applied only to the final hidden layer.
  • Scalability to Complex Visual Tasks: On challenging benchmarks without pre-training (e.g., Split CIFAR-10), while ENNs outperformed baselines, absolute accuracy remains low (~$24\%$), indicating architecture modifications alone cannot fully resolve forgetting in complex end-to-end vision tasks without pre-training or memory mechanisms.

Summary updated

Abstract

Catastrophic forgetting remains a significant challenge to continual learning for decades. While recent works have proposed effective methods to mitigate this problem, they mainly focus on the algorithmic side. Meanwhile, we do not fully understand what architectural properties of neural networks lead to catastrophic forgetting. This study aims to fill this gap by studying the role of activation functions in the training dynamics of neural networks and their impact on catastrophic forgetting. Our study reveals that, besides sparse representations, the gradient sparsity of activation functions also plays an important role in reducing forgetting. Based on this insight, we propose a new class of activation functions, elephant activation functions, that can generate both sparse representations and sparse gradients. We show that by simply replacing classical activation functions with elephant activation functions, we can significantly improve the resilience of neural networks to catastrophic forgetting. Our method has broad applicability and benefits for continual learning in regression, class incremental learning, and reinforcement learning tasks. Specifically, we achieves excellent performance on Split MNIST dataset in just one single pass, without using replay buffer, task boundary information, or pre-training.
BibTeX
@misc{lan2023elephant,
  author = {Qingfeng Lan and A. Rupam Mahmood},
  title = {{Elephant Neural Networks: Born to Be a Continual Learner}},
  year = {2023},
  eprint = {2310.01365},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.01365},
}
Figure