Abstract
Remembering and forgetting mechanisms are two sides of the same coin in a human learning-memory system. Inspired by human brain memory mechanisms, modern machine learning systems have been working to endow machines with lifelong learning capability through better remembering while pushing forgetting as the antagonist to overcome. Nevertheless, this idea might only see half the picture. Increasing research argues that a brain is born to forget; forgetting is a natural and active process for abstract, rich, and flexible representations. This paper presents a learning model by active forgetting mechanism with artificial neural networks. The active forgetting mechanism (AFM) is introduced via a "plug-and-play" forgetting layer (P&PF), consisting of groups of inhibitory neurons with Internal Regulation Strategy (IRS) and External Regulation Strategy (ERS). Experimental studies show that the P&PF offers surprising benefits: self-adaptive structure, strong generalization, long-term learning and memory, and robustness to data and parameter perturbation.
Introduction
In deep learning, "forgetting" is typically viewed as a failure—specifically "catastrophic forgetting," where a model loses prior knowledge when learning new tasks. However, biological systems utilize forgetting as an active, essential process to filter noise, abstract information, and maintain flexibility. Modern artificial neural networks (ANNs) are often over-parameterized and lack this active regulatory mechanism. While tools like Dropout or L1 regularization encourage sparsity, they lack an explicit, biologically inspired mechanism to selectively deactivate neurons to optimize the learning-memory balance.
The Plug-and-Play Forgetting (P&PF) Layer
The authors propose the Active Forgetting Mechanism (AFM), implemented via a Plug-and-Play Forgetting (P&PF) layer. This layer is inserted between standard excitatory layers and utilizes inhibitory neurons governed by two primary strategies:
- External Regulation Strategy (ERS): Inhibitory neurons act as a bottleneck for excitatory signals. By applying a softmax-based soft threshold, the model ensures only the most relevant signals propagate forward.
- Internal Regulation Strategy (IRS): This mimics lateral inhibition, where inhibitory neurons compete. A strong inhibitory neuron suppresses its neighbors, ensuring the forgetting process itself remains sparse and efficient.
To ensure trainability, a shortcut connection is included to prevent gradient vanishing, a common issue when using softmax-based bottlenecks in deep architectures.
Experimental Evaluation
The model was evaluated across several benchmarks to test structural adaptation and memory:
- Fashion MNIST: Tested generalization and neuron importance. The P&PF model achieved a 13% improvement in test accuracy over vanilla SGD, outperforming standard L1 and L2 regularization.
- CIFAR-10: Demonstrated the ability to store information with fewer parameters and faster convergence.
- Permuted MNIST: Used to evaluate lifelong learning. When combined with Elastic Weight Consolidation (EWC), the P&PF layer significantly mitigated catastrophic forgetting compared to standard EWC alone.
Key Insights and Mechanisms
Bimodal Distribution of Importance
In standard networks, neuron importance is often distributed uniformly. The P&PF layer causes importance values to polarize into a bimodal distribution: neurons become either highly critical or completely redundant. This polarization allows for extreme pruning; the model can maintain high accuracy even after removing 80% of its neurons, provided they are removed based on the unimportance defined by the forgetting layer.
Forgetting as a Memory Aid
Counter-intuitively, active forgetting facilitates long-term memory. By clearing non-essential historical memory capacity, the network frees resources for new tasks. Furthermore, because the model learns to ignore insignificant neurons, it exhibits increased robustness to noise in both data and internal parameters.
Conclusion
The P&PF approach achieves efficient online pruning similar to complex offline methods like "Optimal Brain Surgery" but within a single training session. While current implementations use a global receptive field for lateral inhibition, future work may explore local receptive fields for better biological plausibility and computational efficiency. Integrating this mechanism with attention modules or external memory models could further enhance the consolidation of short-term memories into long-term storage.
Summary updated