Pruning of Deep Spiking Neural Networks through Gradient Rewiring

Yanqi Chen, Zhaofei Yu, Wei Fang, Tiejun Huang, Yonghong Tian

Abstract

Spiking Neural Networks (SNNs) have gained significant importance due to their biological plausibility and high energy efficiency on neuromorphic chips. However, these chips are often resource-constrained, making SNN compression essential for practical deployment. Traditional pruning methods, often adapted from artificial neural networks (ANNs), frequently overlook the unique characteristics of SNNs and are typically limited to shallow architectures. In this paper, we propose Gradient Rewiring (Grad R), a joint learning algorithm for connectivity and weights inspired by biological synaptogenesis and synapse elimination. Grad R enables seamless optimization of network structure during training without the need for separate retraining phases. By redefining the gradient for a new synaptic parameter, we facilitate a robust exploration of network structures through the dynamic competition between pruning and regrowth. Our experimental results demonstrate state-of-the-art performance on MNIST and CIFAR-10, achieving minimal accuracy loss even at extreme sparsity levels, such as a ~3.5% drop at 0.73% connectivity. This work highlights the significant redundancy in deep SNNs and provides a path toward highly efficient neuromorphic computing.

Introduction

Spiking Neural Networks (SNNs) represent the "third generation" of neural networks, mimicking the brain's communication via discrete electrical impulses or spikes. This sparse, event-driven activity makes them ideal for low-power neuromorphic hardware. However, the limited memory and compute resources of such hardware necessitate aggressive model compression.

Existing pruning techniques for SNNs face two primary challenges. First, they often employ "one-way pruning," where a connection, once removed, cannot be restored. This limits the network's ability to explore alternative, potentially superior architectures during training. Second, many current methods are optimized for shallow networks and fail to scale effectively to the complexity of deep SNNs. To address these issues, we introduce a unified framework that simultaneously optimizes connectivity and weights, allowing for the discovery of optimal sparse architectures on-the-fly.

Gradient Rewiring (Grad R)

The core of our approach is the Gradient Rewiring (Grad R) framework. Inspired by the brain's ability to form and eliminate synapses, we introduce a hidden synaptic parameter, $\theta$. The actual weight $w$ of a synapse is derived from $\theta$ via a ReLU-like mapping: $w = s \cdot \text{ReLU}(\theta)$, where $s$ is a sign indicator. If $\theta > 0$, the connection is active; if $\theta \leq 0$, the connection is pruned.

Redefining the Gradient

In standard gradient descent, a pruned connection ($w=0$) typically has a zero gradient, effectively "killing" the synapse permanently. Grad R redefines the gradient calculation such that even pruned connections (where $\theta < 0$) receive updates. This allows a negative $\theta$ to eventually cross the threshold and become positive again, "regrowing" a previously deleted connection. This mechanism enables the network to continuously refine its structure throughout the training process.

Redefinition of the gradient for inactive synapses (theta < 0) allows for connection regrowth.

Connectivity Control

To guide the network toward a target sparsity level, we apply a Laplacian prior distribution to $\theta$. This acts as a penalty term, encouraging the network to maintain $\theta$ near a specific value. By adjusting the penalty strength ($\alpha$), researchers can tune the final "budget" of connections to match hardware constraints. Additionally, since the spiking function is non-differentiable, we utilize a surrogate gradient (a shifted ArcTan function) to enable standard backpropagation.

Experimental Evaluation

We evaluated Grad R on two standard benchmarks: MNIST (using a shallow fully-connected SNN) and CIFAR-10 (using a deep convolutional SNN with 6 convolutional and 2 fully-connected layers).

Performance and Sparsity

Grad R demonstrates remarkable resilience to pruning. On CIFAR-10, the model achieved 92.03% accuracy with only ~5% connectivity, representing less than a 1% drop from the unpruned baseline. Even more strikingly, the model maintained 89.32% accuracy at an extreme 0.73% connectivity. This means nearly 99.3% of synapses were removed with only a ~3.5% loss in accuracy.

Performance comparison on MNIST and CIFAR-10, alongside the distribution of theta and pruning/regrowth dynamics.

Dynamics of Rewiring

The success of Grad R stems from the dynamic competition between pruning and regrowth. As shown in the experimental results, the number of pruned and regrown connections eventually reaches an equilibrium, allowing the network to settle into a highly efficient structure. This joint optimization outperforms traditional "train-prune-retrain" cycles and previous rewiring methods like "Deep R," particularly at high sparsity levels.

Discussion and Future Work

Our findings suggest an immense level of redundancy in deep SNNs. The ability to maintain high performance with less than 1% of connections indicates that current models are significantly over-parameterized for these tasks.

While Grad R is highly effective, it does have limitations. The final connectivity is sensitive to hyperparameters like $\alpha$, requiring careful tuning to hit specific hardware budgets. Furthermore, our current implementation excludes Batch Normalization layers from pruning to avoid destabilizing feature channels. Future research will focus on extending pruning to all layer types and measuring real-world energy savings on physical neuromorphic platforms like Intel's Loihi or the SpiNNaker system.

Our code is available at: https://github.com/Yanqi-Chen/Gradient-Rewiring

Summary updated

Abstract

Spiking Neural Networks (SNNs) have been attached great importance due to their biological plausibility and high energy-efficiency on neuromorphic chips. As these chips are usually resource-constrained, the compression of SNNs is thus crucial along the road of practical use of SNNs. Most existing methods directly apply pruning approaches in artificial neural networks (ANNs) to SNNs, which ignore the difference between ANNs and SNNs, thus limiting the performance of the pruned SNNs. Besides, these methods are only suitable for shallow SNNs. In this paper, inspired by synaptogenesis and synapse elimination in the neural system, we propose gradient rewiring (Grad R), a joint learning algorithm of connectivity and weight for SNNs, that enables us to seamlessly optimize network structure without retraining. Our key innovation is to redefine the gradient to a new synaptic parameter, allowing better exploration of network structures by taking full advantage of the competition between pruning and regrowth of connections. The experimental results show that the proposed method achieves minimal loss of SNNs' performance on MNIST and CIFAR-10 dataset so far. Moreover, it reaches a $\sim$3.5% accuracy loss under unprecedented 0.73% connectivity, which reveals remarkable structure refining capability in SNNs. Our work suggests that there exists extremely high redundancy in deep SNNs. Our codes are available at https://github.com/Yanqi-Chen/Gradient-Rewiring.
BibTeX
@inproceedings{chen2021pruning,
  author = {Yanqi Chen and Zhaofei Yu and Wei Fang and Tiejun Huang and Yonghong Tian},
  title = {{Pruning of Deep Spiking Neural Networks through Gradient Rewiring}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=S5KXcpPDJ-M},
}
Figure