Learning by Active Forgetting for Neural Networks

Jian Peng, Xian Sun, Min Deng, Chao Tao, Bo Tang, Wenbo Li, Guohua Wu, Qingzhu, Yu Liu, Tao Lin, Haifeng Li

Abstract

Remembering and forgetting mechanisms are two sides of the same coin in a human learning-memory system. Inspired by human brain memory mechanisms, modern machine learning systems have been working to endow machines with lifelong learning capability through better remembering while pushing forgetting as the antagonist to overcome. Nevertheless, this idea might only see half the picture. Increasing research argues that a brain is born to forget; forgetting is a natural and active process for abstract, rich, and flexible representations. This paper presents a learning model by active forgetting mechanism with artificial neural networks. The active forgetting mechanism (AFM) is introduced via a "plug-and-play" forgetting layer (P&PF), consisting of groups of inhibitory neurons with Internal Regulation Strategy (IRS) and External Regulation Strategy (ERS). Experimental studies show that the P&PF offers surprising benefits: self-adaptive structure, strong generalization, long-term learning and memory, and robustness to data and parameter perturbation.

Introduction

In deep learning, "forgetting" is typically viewed as a failure—specifically "catastrophic forgetting," where a model loses prior knowledge when learning new tasks. However, biological systems utilize forgetting as an active, essential process to filter noise, abstract information, and maintain flexibility. Modern artificial neural networks (ANNs) are often over-parameterized and lack this active regulatory mechanism. While tools like Dropout or L1 regularization encourage sparsity, they lack an explicit, biologically inspired mechanism to selectively deactivate neurons to optimize the learning-memory balance.

The Plug-and-Play Forgetting (P&PF) Layer

The authors propose the Active Forgetting Mechanism (AFM), implemented via a Plug-and-Play Forgetting (P&PF) layer. This layer is inserted between standard excitatory layers and utilizes inhibitory neurons governed by two primary strategies:

  1. External Regulation Strategy (ERS): Inhibitory neurons act as a bottleneck for excitatory signals. By applying a softmax-based soft threshold, the model ensures only the most relevant signals propagate forward.
  2. Internal Regulation Strategy (IRS): This mimics lateral inhibition, where inhibitory neurons compete. A strong inhibitory neuron suppresses its neighbors, ensuring the forgetting process itself remains sparse and efficient.
Schematic of the P&PF layer architecture, illustrating the interaction between excitatory layers, inhibitory neurons, and the gradient-preserving shortcut connection.

To ensure trainability, a shortcut connection is included to prevent gradient vanishing, a common issue when using softmax-based bottlenecks in deep architectures.

Experimental Evaluation

The model was evaluated across several benchmarks to test structural adaptation and memory:

  • Fashion MNIST: Tested generalization and neuron importance. The P&PF model achieved a 13% improvement in test accuracy over vanilla SGD, outperforming standard L1 and L2 regularization.
  • CIFAR-10: Demonstrated the ability to store information with fewer parameters and faster convergence.
  • Permuted MNIST: Used to evaluate lifelong learning. When combined with Elastic Weight Consolidation (EWC), the P&PF layer significantly mitigated catastrophic forgetting compared to standard EWC alone.

Key Insights and Mechanisms

Bimodal Distribution of Importance

In standard networks, neuron importance is often distributed uniformly. The P&PF layer causes importance values to polarize into a bimodal distribution: neurons become either highly critical or completely redundant. This polarization allows for extreme pruning; the model can maintain high accuracy even after removing 80% of its neurons, provided they are removed based on the unimportance defined by the forgetting layer.

3D visualization of neuron activation evolution from initialization through 15 epochs, showing the transition toward a structured, sparse state.

Forgetting as a Memory Aid

Counter-intuitively, active forgetting facilitates long-term memory. By clearing non-essential historical memory capacity, the network frees resources for new tasks. Furthermore, because the model learns to ignore insignificant neurons, it exhibits increased robustness to noise in both data and internal parameters.

Conclusion

The P&PF approach achieves efficient online pruning similar to complex offline methods like "Optimal Brain Surgery" but within a single training session. While current implementations use a global receptive field for lateral inhibition, future work may explore local receptive fields for better biological plausibility and computational efficiency. Integrating this mechanism with attention modules or external memory models could further enhance the consolidation of short-term memories into long-term storage.

Summary updated

Abstract

Remembering and forgetting mechanisms are two sides of the same coin in a human learning-memory system. Inspired by human brain memory mechanisms, modern machine learning systems have been working to endow machine with lifelong learning capability through better remembering while pushing the forgetting as the antagonist to overcome. Nevertheless, this idea might only see the half picture. Up until very recently, increasing researchers argue that a brain is born to forget, i.e., forgetting is a natural and active process for abstract, rich, and flexible representations. This paper presents a learning model by active forgetting mechanism with artificial neural networks. The active forgetting mechanism (AFM) is introduced to a neural network via a "plug-and-play" forgetting layer (P\&PF), consisting of groups of inhibitory neurons with Internal Regulation Strategy (IRS) to adjust the extinction rate of themselves via lateral inhibition mechanism and External Regulation Strategy (ERS) to adjust the extinction rate of excitatory neurons via inhibition mechanism. Experimental studies have shown that the P\&PF offers surprising benefits: self-adaptive structure, strong generalization, long-term learning and memory, and robustness to data and parameter perturbation. This work sheds light on the importance of forgetting in the learning process and offers new perspectives to understand the underlying mechanisms of neural networks.
BibTeX
@misc{peng2021learning,
  author = {Jian Peng and Xian Sun and Min Deng and Chao Tao and Bo Tang and Wenbo Li and Guohua Wu and Qingzhu and Yu Liu and Tao Lin and Haifeng Li},
  title = {{Learning by Active Forgetting for Neural Networks}},
  year = {2021},
  eprint = {2111.10831},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.10831},
}
Figure