TL;DR
Most continual learning methods rely on complex algorithmic workarounds like replay buffers or task boundaries to mitigate catastrophic forgetting. This paper reveals that network architecture—specifically activation functions—plays a massive role in forgetting, showing that neural networks need both sparse representations and sparse gradients. To achieve this, the authors introduce Elephant Activation Functions, bell-shaped activations that grant deep networks local elasticity, enabling effective continual learning across regression, classification, and reinforcement learning with minimal or no replay buffers.
What Problem Does This Paper Address?
When artificial neural networks learn sequentially from non-stationary (non-iid) data, they suffer from catastrophic forgetting: updating parameters on new data erases previously acquired knowledge. While existing research has focused on external algorithmic fixes (such as storing past examples in replay buffers, regularizing important weights like EWC, or dynamically isolating sub-networks), the fundamental question of why standard architectures forget so aggressively remains under-explored.
The authors investigate the training dynamics of neural networks through the lens of the Neural Tangent Kernel (NTK) and local elasticity—the ability of a network to update its prediction on a new input without altering predictions on dissimilar inputs.
Methods
Analyzing the NTK of a single-hidden-layer network $f_{\mathbf{w}}(\mathbf{x}) = \mathbf{u}^\top \sigma(\mathbf{V}\mathbf{x} + \mathbf{b})$, the parameter update under gradient descent reveals two interaction terms: $$\langle \nabla_{\mathbf{w}} f(\mathbf{x}), \nabla_{\mathbf{w}} f(\mathbf{x}_t) \rangle = \sigma(\mathbf{V}\mathbf{x} + \mathbf{b})^\top \sigma(\mathbf{V}\mathbf{x}_t + \mathbf{b}) + \mathbf{u}^\top\mathbf{u}(\mathbf{x}^\top \mathbf{x}_t + 1)\sigma'(\mathbf{V}\mathbf{x} + \mathbf{b})^\top \sigma'(\mathbf{V}\mathbf{x}_t + \mathbf{b})$$
- The Failure of Sparse Representations Alone: Prior works often rely solely on sparse feature activations ($\sigma(\cdot) \approx \mathbf{0}$). However, the second term involves the gradients of the activation function ($\sigma'(\cdot)$). If the derivative $\sigma'$ is dense, updating on $\mathbf{x}_t$ still causes non-local weight changes, breaking local elasticity.
- Elephant Activation Function: The authors introduce a bell-shaped activation function designed to enforce sparsity in both function values and derivative values: $$\text{Elephant}(x) = \frac{1}{1 + \left|\frac{x}{a}\right|^d}$$
- $a$ modulates the function width (controlling representation sparsity).
- $d$ controls the steepness of the slope (controlling gradient sparsity).
- Elephant Neural Networks (ENNs): In practice, standard activations (like ReLU) in the final hidden layer are replaced with Elephant functions (e.g., EMLP, ECNN). Bias terms in this layer are initialized evenly across an interval to ensure feature diversity.
Datasets
The authors evaluate ENNs across three distinct continual learning paradigms:
- Streaming 1D Regression: Continuous streaming approximation of a sine wave $y = \sin(\pi x)$ over $x \in [0, 2]$ in a single pass ($n = 200$ points, evaluated on 1,000 evenly spaced test points).
- Class-Incremental Learning: Evaluated under a strict streaming setting (single pass, no replay buffer, no task boundaries, no pre-training):
- Split MNIST (5 tasks of 2 classes each).
- Split CIFAR-10 (5 tasks of 2 classes each).
- Pre-trained embedding variants: Split Embedding CIFAR-10, Split Embedding CIFAR-100, and Split Embedding Tiny ImageNet (using frozen ConvMixer representations).
- Reinforcement Learning: Four continuous-state, discrete-action control benchmarks from OpenAI Gym and PyGame Learning Environment under extreme buffer constraints: MountainCar-v0, Acrobot-v1, Catcher, and Pixelcopter.
Key Insights and Ideas
- Sparse representations need sparse gradients: In linear models, representation sparsity suffices for local updates. In non-linear deep networks, gradients of the feature extractor propagate across the network unless $\sigma'(\cdot)$ is also sparse.
- True point-wise editing: Standard MLPs exhibit global generalization—updating a prediction at $x = 1.5$ drastically shifts predictions across the entire input domain. ENNs exhibit strong local elasticity, altering predictions exclusively in a narrow window around the updated point.
- Architectural fixes outperform algorithmic patches: A standard network augmented with Streaming EWC yields only modest gains, whereas switching the architecture to an Elephant network provides substantially higher resilience to forgetting in single-pass settings.
Key Results
- Streaming Regression: On the streaming sine task, classical MLPs (ReLU, Sigmoid, Tanh, ELU) fail with test MSEs ranging from $0.446$ to $0.473$. Sparse-representation networks (SR-NN) achieve $0.406$. EMLP achieves an MSE of $0.0081 \pm 0.0009$.
- Class-Incremental Learning (Split MNIST):
- With a 10K-neuron hidden layer in a single pass without task boundaries: standard MLP reaches $62.1\%$ and MLP + Streaming EWC reaches $60.9\%$.
- EMLP achieves $80.2 \pm 0.2\%$, and ECNN achieves $85.0 \pm 0.4\%$, approaching FlyModel ($91\%$), which relies on explicit task boundary awareness.
- On Split CIFAR-10 (10K neurons), ECNN achieves $24.3\%$ vs. CNN's $15.1\%$ and CNN + Streaming EWC's $17.9\%$.
- Reinforcement Learning: In DQN agent training, an EMLP using a tiny replay buffer ($m = 32$) outperforms standard MLPs with the same buffer size across all 4 environments and matches or rivals standard MLPs using a massive replay buffer ($m = 10,000$) on MountainCar, Acrobot, and Catcher.
Limitations and Future Work
- Hyperparameter Sensitivity: There is currently no analytical method to set the function width $a$ or the bias initialization spread $\sigma_{\text{bias}}$; they must be tuned empirically based on the input dimension and data distribution.
- Loss of Plasticity in Deep Layers: Initial experiments showed that replacing activations across all layers hurt performance, likely due to optimization difficulties and loss of plasticity. Consequently, Elephant activations were applied only to the final hidden layer.
- Scalability to Complex Visual Tasks: On challenging benchmarks without pre-training (e.g., Split CIFAR-10), while ENNs outperformed baselines, absolute accuracy remains low (~$24\%$), indicating architecture modifications alone cannot fully resolve forgetting in complex end-to-end vision tasks without pre-training or memory mechanisms.
Summary updated