Abstract
Spiking Neural Networks (SNNs) have gained significant importance due to their biological plausibility and high energy efficiency on neuromorphic chips. However, these chips are often resource-constrained, making SNN compression essential for practical deployment. Traditional pruning methods, often adapted from artificial neural networks (ANNs), frequently overlook the unique characteristics of SNNs and are typically limited to shallow architectures. In this paper, we propose Gradient Rewiring (Grad R), a joint learning algorithm for connectivity and weights inspired by biological synaptogenesis and synapse elimination. Grad R enables seamless optimization of network structure during training without the need for separate retraining phases. By redefining the gradient for a new synaptic parameter, we facilitate a robust exploration of network structures through the dynamic competition between pruning and regrowth. Our experimental results demonstrate state-of-the-art performance on MNIST and CIFAR-10, achieving minimal accuracy loss even at extreme sparsity levels, such as a ~3.5% drop at 0.73% connectivity. This work highlights the significant redundancy in deep SNNs and provides a path toward highly efficient neuromorphic computing.
Introduction
Spiking Neural Networks (SNNs) represent the "third generation" of neural networks, mimicking the brain's communication via discrete electrical impulses or spikes. This sparse, event-driven activity makes them ideal for low-power neuromorphic hardware. However, the limited memory and compute resources of such hardware necessitate aggressive model compression.
Existing pruning techniques for SNNs face two primary challenges. First, they often employ "one-way pruning," where a connection, once removed, cannot be restored. This limits the network's ability to explore alternative, potentially superior architectures during training. Second, many current methods are optimized for shallow networks and fail to scale effectively to the complexity of deep SNNs. To address these issues, we introduce a unified framework that simultaneously optimizes connectivity and weights, allowing for the discovery of optimal sparse architectures on-the-fly.
Gradient Rewiring (Grad R)
The core of our approach is the Gradient Rewiring (Grad R) framework. Inspired by the brain's ability to form and eliminate synapses, we introduce a hidden synaptic parameter, $\theta$. The actual weight $w$ of a synapse is derived from $\theta$ via a ReLU-like mapping: $w = s \cdot \text{ReLU}(\theta)$, where $s$ is a sign indicator. If $\theta > 0$, the connection is active; if $\theta \leq 0$, the connection is pruned.
Redefining the Gradient
In standard gradient descent, a pruned connection ($w=0$) typically has a zero gradient, effectively "killing" the synapse permanently. Grad R redefines the gradient calculation such that even pruned connections (where $\theta < 0$) receive updates. This allows a negative $\theta$ to eventually cross the threshold and become positive again, "regrowing" a previously deleted connection. This mechanism enables the network to continuously refine its structure throughout the training process.
Connectivity Control
To guide the network toward a target sparsity level, we apply a Laplacian prior distribution to $\theta$. This acts as a penalty term, encouraging the network to maintain $\theta$ near a specific value. By adjusting the penalty strength ($\alpha$), researchers can tune the final "budget" of connections to match hardware constraints. Additionally, since the spiking function is non-differentiable, we utilize a surrogate gradient (a shifted ArcTan function) to enable standard backpropagation.
Experimental Evaluation
We evaluated Grad R on two standard benchmarks: MNIST (using a shallow fully-connected SNN) and CIFAR-10 (using a deep convolutional SNN with 6 convolutional and 2 fully-connected layers).
Performance and Sparsity
Grad R demonstrates remarkable resilience to pruning. On CIFAR-10, the model achieved 92.03% accuracy with only ~5% connectivity, representing less than a 1% drop from the unpruned baseline. Even more strikingly, the model maintained 89.32% accuracy at an extreme 0.73% connectivity. This means nearly 99.3% of synapses were removed with only a ~3.5% loss in accuracy.
Dynamics of Rewiring
The success of Grad R stems from the dynamic competition between pruning and regrowth. As shown in the experimental results, the number of pruned and regrown connections eventually reaches an equilibrium, allowing the network to settle into a highly efficient structure. This joint optimization outperforms traditional "train-prune-retrain" cycles and previous rewiring methods like "Deep R," particularly at high sparsity levels.
Discussion and Future Work
Our findings suggest an immense level of redundancy in deep SNNs. The ability to maintain high performance with less than 1% of connections indicates that current models are significantly over-parameterized for these tasks.
While Grad R is highly effective, it does have limitations. The final connectivity is sensitive to hyperparameters like $\alpha$, requiring careful tuning to hit specific hardware budgets. Furthermore, our current implementation excludes Batch Normalization layers from pruning to avoid destabilizing feature channels. Future research will focus on extending pruning to all layer types and measuring real-world energy savings on physical neuromorphic platforms like Intel's Loihi or the SpiNNaker system.
Our code is available at: https://github.com/Yanqi-Chen/Gradient-Rewiring
Summary updated