Introduction
Deep learning models typically require massive datasets to generalize effectively, a stark contrast to the human brain's ability to learn from sparse examples. This research explores a biologically inspired "sleep-like" unsupervised replay phase designed to enhance Artificial Neural Networks (ANNs) when training data is scarce or imbalanced. By mapping an ANN to a Spiking Neural Network (SNN) and applying local plasticity rules, the authors demonstrate significant performance gains in low-data regimes.
Problem Statement: The Data Hunger of ANNs
Artificial Neural Networks are notoriously data-hungry. In "low-data" scenarios or when datasets are heavily imbalanced, ANNs are prone to overfitting and poor generalization. Biological systems, however, utilize sleep to consolidate memories. During sleep, the brain spontaneously replays learned patterns, using local synaptic plasticity to stabilize critical connections. This paper introduces a Sleep Replay Consolidation (SRC) algorithm to bridge this gap, allowing models to learn more effectively from limited or skewed data without relying on extensive data augmentation.
Methodology: The Sleep Replay Consolidation (SRC) Pipeline
The researchers developed a multi-step pipeline to simulate the cognitive benefits of sleep:
- Initial Training: A standard fully-connected ANN with two hidden layers is trained on a small data subset using traditional backpropagation.
- Sleep Transition: The trained ANN is mapped onto a Spiking Neural Network (SNN) with an identical architecture.
- Unsupervised Replay (SRC): The SNN is driven by "Poisson spiking input," representing average training inputs. During this phase, the network employs local Hebbian-type plasticity without seeing new data. Connections are strengthened if a pre-synaptic neuron firing is followed by a post-synaptic firing, and weakened otherwise.
- Remapping and Fine-tuning: The SNN is mapped back to an ANN. For datasets exceeding 10% of the total available data, a brief fine-tuning session on the original training data is conducted to optimize performance.
Experimental Setup
The study utilized two benchmark computer vision datasets:
- MNIST: 70,000 handwritten digits (0-9).
- Fashion MNIST (FMNIST): 70,000 grayscale images of clothing items, representing a more complex classification task. Performance was evaluated across training fractions ranging from 0.1% to 100%.
Key Results and Discussion
The SRC algorithm yielded substantial improvements, particularly in data-constrained environments.
Performance in Low-Data Regimes
As shown in the figures above, for models trained on only 0.5% to 10% of the datasets, the sleep phase provided a 20-30% absolute increase in accuracy. The SRC algorithm effectively helped the network "unlearn" class biases common in low-data scenarios, leading to more balanced confusion matrices.
Sparsity and Structural Refinement
Analysis of synaptic weights revealed that the sleep phase increased the sparsity of the network's responses. By strengthening critical synapses and weakening others, the replay phase helped the model focus on the most salient features of the data.
The "Over-Sleep" Effect and Recovery
Interestingly, for models with ample data (over 10%), the unsupervised sleep phase alone caused a slight performance dip (10-15%). However, this was successfully mitigated by a brief fine-tuning step, which allowed these models to maintain high baseline accuracy while benefiting from the structural refinements of the sleep phase.
Conclusion and Future Work
This work provides a proof-of-concept for using unsupervised, biologically inspired replay to improve ANN performance. While the current study focused on simple fully-connected architectures and image classification, it opens doors for future research into:
- Scaling SRC to complex architectures like CNNs or Transformers.
- Exploring diverse plasticity rules.
- Applying these techniques to more complex, real-world datasets.
The findings suggest that synaptic weight dynamics during sleep-like states can significantly enhance memory consolidation and generalization in artificial systems.
Summary updated