Sleep-Like Unsupervised Replay Improves Performance When Data Are Limited or Unbalanced (Student Abstract)

Anthony Bazhenov, Pahan Dewasurendra, Giri P. Krishnan, Jean Erik DelanoisAAAI 2024 · DOI

Introduction

Deep learning models typically require massive datasets to generalize effectively, a stark contrast to the human brain's ability to learn from sparse examples. This research explores a biologically inspired "sleep-like" unsupervised replay phase designed to enhance Artificial Neural Networks (ANNs) when training data is scarce or imbalanced. By mapping an ANN to a Spiking Neural Network (SNN) and applying local plasticity rules, the authors demonstrate significant performance gains in low-data regimes.

Problem Statement: The Data Hunger of ANNs

Artificial Neural Networks are notoriously data-hungry. In "low-data" scenarios or when datasets are heavily imbalanced, ANNs are prone to overfitting and poor generalization. Biological systems, however, utilize sleep to consolidate memories. During sleep, the brain spontaneously replays learned patterns, using local synaptic plasticity to stabilize critical connections. This paper introduces a Sleep Replay Consolidation (SRC) algorithm to bridge this gap, allowing models to learn more effectively from limited or skewed data without relying on extensive data augmentation.

Methodology: The Sleep Replay Consolidation (SRC) Pipeline

The researchers developed a multi-step pipeline to simulate the cognitive benefits of sleep:

  1. Initial Training: A standard fully-connected ANN with two hidden layers is trained on a small data subset using traditional backpropagation.
  2. Sleep Transition: The trained ANN is mapped onto a Spiking Neural Network (SNN) with an identical architecture.
  3. Unsupervised Replay (SRC): The SNN is driven by "Poisson spiking input," representing average training inputs. During this phase, the network employs local Hebbian-type plasticity without seeing new data. Connections are strengthened if a pre-synaptic neuron firing is followed by a post-synaptic firing, and weakened otherwise.
  4. Remapping and Fine-tuning: The SNN is mapped back to an ANN. For datasets exceeding 10% of the total available data, a brief fine-tuning session on the original training data is conducted to optimize performance.

Experimental Setup

The study utilized two benchmark computer vision datasets:

  • MNIST: 70,000 handwritten digits (0-9).
  • Fashion MNIST (FMNIST): 70,000 grayscale images of clothing items, representing a more complex classification task. Performance was evaluated across training fractions ranging from 0.1% to 100%.

Key Results and Discussion

The SRC algorithm yielded substantial improvements, particularly in data-constrained environments.

Accuracy vs. Fraction of Total Data for MNIST (A) and FMNIST (B) across different training regimes.

Performance in Low-Data Regimes

As shown in the figures above, for models trained on only 0.5% to 10% of the datasets, the sleep phase provided a 20-30% absolute increase in accuracy. The SRC algorithm effectively helped the network "unlearn" class biases common in low-data scenarios, leading to more balanced confusion matrices.

Sparsity and Structural Refinement

Analysis of synaptic weights revealed that the sleep phase increased the sparsity of the network's responses. By strengthening critical synapses and weakening others, the replay phase helped the model focus on the most salient features of the data.

The "Over-Sleep" Effect and Recovery

Interestingly, for models with ample data (over 10%), the unsupervised sleep phase alone caused a slight performance dip (10-15%). However, this was successfully mitigated by a brief fine-tuning step, which allowed these models to maintain high baseline accuracy while benefiting from the structural refinements of the sleep phase.

Conclusion and Future Work

This work provides a proof-of-concept for using unsupervised, biologically inspired replay to improve ANN performance. While the current study focused on simple fully-connected architectures and image classification, it opens doors for future research into:

  • Scaling SRC to complex architectures like CNNs or Transformers.
  • Exploring diverse plasticity rules.
  • Applying these techniques to more complex, real-world datasets.

The findings suggest that synaptic weight dynamics during sleep-like states can significantly enhance memory consolidation and generalization in artificial systems.

Summary updated

BibTeX
@inproceedings{bazhenov2024sleep,
  author = {Anthony Bazhenov and Pahan Dewasurendra and Giri P. Krishnan and Jean Erik Delanois},
  title = {{Sleep-Like Unsupervised Replay Improves Performance When Data Are Limited or Unbalanced (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30420},
}
Figure