Unifying Importance Based Regularisation Methods for Continual Learning

Frederik BenzingAISTATS 2022

Introduction

Artificial neural networks are notoriously prone to catastrophic forgetting, where learning new tasks leads to the erasure of previously acquired knowledge. Continual Learning (CL) aims to mitigate this through various strategies, most notably importance-based regularization. This family of methods identifies critical parameters for a given task and penalizes their modification during subsequent training. Three primary algorithms have historically dominated this space: Elastic Weight Consolidation (EWC), Synaptic Intelligence (SI), and Memory Aware Synapses (MAS). Despite their shared goal, these methods were derived from disparate theoretical foundations—Bayesian inference, path integrals, and sensitivity heuristics, respectively. This work unifies these "Big Three" methods by demonstrating they are mathematically convergent, specifically through their relationship with the Fisher Information Matrix.

Deconstructing Synaptic Intelligence

A core contribution of this research is the theoretical and empirical deconstruction of Synaptic Intelligence (SI). SI identifies parameter importance by calculating a path integral of the gradient along the optimization trajectory. However, this study reveals that SI’s importance measure is heavily dominated by a statistical bias.

Summed importance values over training iterations, highlighting the significant bias in standard SI compared to unbiased versions.

As shown in the figure above, the "biased" SI importance grows significantly faster than its unbiased counterparts (SIU) or the full gradient. This bias arises because SI uses the same minibatch of data to both update weights and estimate their importance. The resulting "noise" in the gradient is not a nuisance but rather the primary mechanism through which SI functions.

The Fisher Connection and SOS

By analyzing this productive bias, the author proves that SI’s importance measure is approximately equal to the square root of the Fisher Information. A similar derivation shows that MAS, under the assumption of model confidence, also approximates this square root. In contrast, EWC utilizes the Fisher Information directly. This insight provides a unified theoretical framework: EWC uses the Fisher, while SI and MAS use its square root.

To leverage this discovery, the author proposes Second-Order Synapses (SOS). Unlike SI, which relies on accidental stochastic bias, SOS explicitly calculates the square root of the Fisher Information using the second moment of gradients.

Pearson correlation between SI/SIB/SIU and SOS (square root of Fisher) across 10 tasks of Permuted MNIST.

Empirical validation on Permuted MNIST confirms this link, showing a Pearson correlation coefficient near 1.0 between SI (and its biased variant SIB) and the SOS importance values.

Experimental Results

The proposed SOS algorithm was evaluated against standard benchmarks including Permuted MNIST, Split CIFAR 10/100, and Tiny ImageNet.

  • Robustness to Batch Size: Because SI depends on gradient noise, its performance degrades significantly with large batch sizes. On Split CIFAR with a batch size of 2048, SI’s accuracy fell to 70.0%, whereas SOS remained stable at 74.4%.
  • Large-Scale Performance: On Tiny ImageNet using VGG architectures, SOS consistently outperformed SI. For a "Small" VGG model, SOS achieved 44.3% accuracy, nearly doubling SI’s 25.1%.
  • Efficiency: SOS offers superior computational efficiency as it can accumulate importance measures "online" during training, avoiding the need for an additional data pass required by EWC or MAS.

Conclusion

This work provides a rigorous theoretical unification of the most influential regularization methods in Continual Learning. By identifying the hidden bias in SI and its connection to the Fisher Information, the author not only clarifies why these disparate methods work but also introduces SOS—a more robust and efficient alternative. While the practical superiority of the square root of the Fisher over the Fisher itself remains a subject for future Bayesian investigation, SOS represents a significant step toward principled and scalable Continual Learning.

Summary updated

BibTeX
@inproceedings{benzing2022unifying,
  author = {Frederik Benzing},
  title = {{Unifying Importance Based Regularisation Methods for Continual Learning}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2022},
}
Figure