Adaptive Sparse Structure Development with Pruning and Regeneration for Spiking Neural Networks

Bing Han, Feifei Zhao, Yi Zeng, Wenxuan PanInformation Sciences 2022 · OpenReview

TL;DR

This paper introduces SD-SNN, a novel method for developing compact and efficient Spiking Neural Networks (SNNs) by mimicking brain development. It uses dendritic spine plasticity to adaptively prune and regenerate synapses, achieving superior performance and significant energy reduction across various datasets.

What Problem Does This Paper Address?

Spiking Neural Networks (SNNs) hold great promise for energy-efficient computation due to their biological plausibility. However, their practical application is hindered by their often complex and fixed structures, which lead to high energy consumption, especially in deep SNNs. While existing methods attempt to compress SNNs, they often lack inspiration from the brain's own developmental plasticity mechanisms. This gap means current SNN compression techniques may not fully leverage the inherent advantages of SNNs or achieve optimal structural efficiency and performance. The authors aim to bridge this gap by proposing a method that dynamically develops SNN structures, mirroring how the brain prunes and refines its neural connections.

Methods

The core of the SD-SNN method lies in three key brain-inspired developmental mechanisms:

  • Dendritic Spine Plasticity-based Synaptic Constraint: This mechanism mimics how dendritic spines, the primary sites of excitatory synapses, dynamically adjust their efficacy. The SD-SNN imposes positive and negative boundaries on synaptic weights, preventing them from exceeding certain limits. These boundaries are adaptively updated based on synaptic activity, allowing the network to strengthen important connections and weaken redundant ones.
  • Neuronal Pruning: Inspired by the brain's process of eliminating less active neurons, this method prunes neurons based on their activity level. Neuron importance is assessed by summing the "importance" of their incoming synapses, where importance is defined by the range of synaptic weight boundaries. This ensures that neurons with consistently weak or near-zero incoming signals are identified for removal.
  • Synaptic Regeneration: To counteract the potential for over-pruning and performance degradation, SD-SNN incorporates a synaptic regeneration mechanism. This allows pruned synapses to be reactivated and updated if they show signs of importance (e.g., large gradient updates over several epochs). This process helps repair damage caused by aggressive pruning and maintain network stability.

Crucially, the pruning and regeneration rates are adaptive. Inspired by neurotrophic hypotheses, the pruning rate is dynamically modulated based on the number of neurons in adjacent layers, aiming for a balanced network structure. The regeneration rate also increases with training epochs, allowing for more opportunities to recover potentially important but pruned connections.

Datasets

The SD-SNN model was evaluated on a variety of datasets:

  • Spatial Datasets:
    • MNIST
    • CIFAR-10
  • Temporal Neuromorphic Datasets:
    • N-MNIST (neuromorphic version of MNIST)
    • DVS-Gesture

Key Insights and Ideas

  • Brain Development as a Blueprint for SNN Compression: The paper strongly advocates for drawing inspiration from biological brain development for SNN compression, moving beyond simply adapting DNN techniques. This approach leads to more "natural" and efficient SNN structures.
  • The Power of Adaptive Dynamics: The adaptive nature of both pruning and regeneration rates is a critical insight. Instead of fixed parameters, allowing these rates to change dynamically during training leads to more robust and effective compression, mirroring the brain's flexible developmental processes.
  • Synaptic Constraint as a Foundation: The dendritic spine plasticity-based synaptic constraint acts as a crucial first step, preparing the network for pruning by identifying and limiting redundant synapses. This lays the groundwork for more effective neuronal pruning.
  • Regeneration as a Safety Net: Synaptic regeneration is shown to be vital for preventing over-pruning and maintaining performance. It acts as a crucial mechanism to repair potential damage and ensure that important connections are not permanently lost.
  • Biological Plausibility and Performance Synergy: The proposed SD-SNN model not only achieves high performance but also demonstrates biological plausibility, suggesting a path towards more biologically realistic and efficient artificial neural networks.

Key Results

  • MNIST: Achieved 99.51% accuracy with a 49.83% compression rate, showing a 0.05% accuracy improvement over the baseline.
  • DVS-Gesture: Achieved 98.20% accuracy with a 55.50% compression rate, demonstrating a 1.09% improvement.
  • N-MNIST: Achieved 99.53% accuracy with a 58.62% compression rate.
  • CIFAR-10: Achieved 94.10% accuracy with a 37.44% compression rate.
  • The method consistently outperforms existing SNN compression techniques across various datasets, often achieving higher accuracy at comparable or even greater compression rates.
  • The adaptive nature of the pruning and regeneration rates leads to a stable network structure and performance, even with different initial pruning rates.

Limitations and Future Work

  • Tunable Hyperparameters: The paper acknowledges that parameters like the number threshold $T_{num}$ for synaptic constraint and the constants $\alpha, \beta, \gamma$ for dynamic rates are tunable hyperparameters. While the paper shows optimal values for $T_{num}$ (18 for MNIST/N-MNIST), further research could explore more automated methods for setting these.
  • Computational Cost: While SNNs are generally more energy-efficient, the training process involving dynamic adaptation and regeneration might introduce computational overhead. The paper doesn't explicitly quantify this training overhead compared to static methods.
  • Future Work: The authors suggest that their SD-SNN model provides a foundation for further exploration of brain developmental mechanisms in SNNs. Future work could involve investigating other developmental processes and their integration into SNN architectures. The availability of code is also a positive step for future research.

Summary updated

Abstract

Spiking Neural Networks (SNNs) are more biologically plausible and computationally efficient. Therefore, SNNs have the natural advantage of drawing the sparse structural plasticity of brain development to alleviate the energy problems of deep neural networks caused by their complex and fixed structures. However, previous SNNs compression works are lack of in-depth inspiration from the brain development plasticity mechanism. This paper proposed a novel method for the adaptive structural development of SNN (SD-SNN), introducing dendritic spine plasticity-based synaptic constraint, neuronal pruning and synaptic regeneration. We found that synaptic constraint and neuronal pruning can detect and remove a large amount of redundancy in SNNs, coupled with synaptic regeneration can effectively prevent and repair over-pruning. Moreover, inspired by the neurotrophic hypothesis, neuronal pruning rate and synaptic regeneration rate were adaptively adjusted during the learning-while-pruning process, which eventually led to the structural stability of SNNs. Experimental results on spatial (MNIST, CIFAR-10) and temporal neuromorphic (N-MNIST, DVS-Gesture) datasets demonstrate that our method can flexibly learn appropriate compression rate for various tasks and effectively achieve superior performance while massively reducing the network energy consumption. Specifically, for the spatial MNIST dataset, our SD-SNN achieves 99.51\% accuracy at the pruning rate 49.83\%, which has a 0.05\% accuracy improvement compared to the baseline without compression. For the neuromorphic DVS-Gesture dataset, 98.20\% accuracy with 1.09\% improvement is achieved by our method when the compression rate reaches 55.50\%.
BibTeX
@article{han2022adaptive,
  author = {Bing Han and Feifei Zhao and Yi Zeng and Wenxuan Pan},
  title = {{Adaptive Sparse Structure Development with Pruning and Regeneration for Spiking Neural Networks}},
  journal = {Information Sciences},
  year = {2022},
  url = {https://openreview.net/forum?id=j2hsiurQJB},
}
Figure