Abstract
Catastrophic forgetting occurs when artificial neural networks overwrite knowledge of previous tasks while learning new ones. We introduce Neuron-level Plasticity Control (NPC), which shifts the focus of consolidation from individual connection weights to the neurons themselves. By dynamically adjusting per-neuron learning rates based on a Taylor expansion importance criterion, NPC preserves critical functional units. Our method achieves significantly higher accuracy on benchmarks like iCIFAR100 (72.43%) and demonstrates superior memory efficiency by maintaining constant overhead per task.
Introduction
In continual learning, training a model on a sequence of tasks often leads to the degradation of performance on earlier tasks, a phenomenon known as catastrophic forgetting. Traditional approaches, such as Elastic Weight Consolidation (EWC), attempt to mitigate this by penalizing changes to important connection weights. However, because weights in deep networks are highly correlated and hierarchical, protecting isolated weights is often insufficient. A single neuron's output depends on the collective state of its incoming connections; thus, changing even "unimportant" weights can disrupt the neuron's learned function.
Methodology: Neuron-level Plasticity Control (NPC)
NPC addresses the limitations of weight-level methods by treating the neuron (or CNN filter) as the fundamental unit of knowledge. The core principle is that protecting a neuron's entire input set is more effective than protecting individual weights.
Importance Evaluation and Plasticity Control
NPC employs a normalized Taylor expansion criterion to estimate neuron importance, utilizing gradients calculated during standard backpropagation for efficiency. These importance values are maintained across tasks using Exponential Moving Averages (EMA).
Instead of adding a regularization term to the global loss function, NPC directly modifies the learning rate ($\eta_i$) for each neuron. By solving an optimization problem that balances stability (preserving old knowledge) and plasticity (learning new tasks), NPC assigns near-zero learning rates to critical neurons, effectively freezing them, while allowing less important neurons to adapt to new data. Additionally, the authors utilize Instance Normalization to handle the distribution shifts inherent in sequential task learning.
Experimental Results
NPC was evaluated on three major benchmarks: iMNIST, iCIFAR100, and a Heterogeneous Sequence (MNIST, fMNIST, EMNIST, and smallNORB).
On the iCIFAR100 benchmark, NPC achieved an average validation accuracy of 72.43%, significantly outperforming traditional methods which typically ranged between 50% and 60%. In the iMNIST experiment, NPC demonstrated near-perfect retention with 99.63% average accuracy. The method also proved robust to domain shifts in the heterogeneous sequence, particularly when encountering the smallNORB dataset, where other models saw significant performance drops. Theoretical validation was further provided by visualizing neuron activation changes, confirming that important neurons remained stable under NPC while learning subsequent tasks.
Memory and Computational Efficiency
Unlike weight-level methods that require storing importance and optimal values for every connection per task, NPC only stores one importance value per neuron. This results in a constant memory overhead, making it highly scalable for long sequences of tasks.
Conclusion
Neuron-level Plasticity Control represents a significant advancement in overcoming catastrophic forgetting. By shifting the granularity of consolidation from weights to neurons, NPC provides a more stable foundation for continual learning without the gradient interference common in regularization-based methods. Future work will explore strategies for minimal capacity usage to further extend the network's ability to learn indefinitely long task sequences.
Summary updated