Abstract
Deep Spiking Neural Networks (SNNs) are promising for low-power neuromorphic computing but are hampered by high inference latency and training instabilities during compression. This paper introduces Spike-Thrift, a two-stage framework designed to optimize SNNs for both sparsity and energy efficiency. By employing Attention-Guided Compression (AGC) to stabilize pruning in networks lacking Batch Normalization and subsequent sparse-learning fine-tuning, Spike-Thrift achieves state-of-the-art compression ratios (up to 33.4x) and significant energy savings (up to 38.7x) over traditional artificial neural networks (ANNs).
Introduction
Spiking Neural Networks (SNNs) mimic the brain's event-driven communication using discrete binary spikes. While inherently energy-efficient on specialized hardware, SNNs suffer from two primary issues: the "Training Gap," where converting trained ANNs to SNNs requires excessive time steps, and the "Compression Paradox," where standard pruning techniques destabilize networks that lack Batch Normalization (BN).
As illustrated in the figure above, networks without BN exhibit highly concentrated gradient magnitudes, which can lead to gradient explosion or vanishing during pruning. Spike-Thrift addresses these challenges by using attention mechanisms to guide the compression process.
Methodology
The Spike-Thrift pipeline consists of two distinct stages aimed at structural and operational efficiency.
Stage 1: Attention-Guided Compression (AGC)
To prune models without the stabilizing effect of Batch Normalization, the authors utilize a "meta-model" (a dense teacher network). An attention-transfer loss term is introduced to ensure the sparse "student" model mimics the critical feature maps of the teacher. This attention-guided approach provides the necessary gradient stability to prune the network effectively before it is converted into the spiking domain.
Stage 2: Sparse-Learning SNN Training
Following compression, the model is converted into an SNN. To recover accuracy and minimize inference latency, the authors apply a sparse-learning (SL) strategy. This involves fine-tuning the SNN using surrogate gradients to handle the non-differentiable nature of spikes. By updating only the non-zero weights, the network learns to maintain high performance with significantly fewer time steps ($T$), directly reducing energy consumption.
Experimental Results
The framework was evaluated on CIFAR-10, CIFAR-100, and Tiny-ImageNet using VGG and ResNet architectures.
- Compression and Accuracy: Spike-Thrift achieved a 33.4x compression ratio on VGG16 for CIFAR-10 with minimal accuracy loss, outperforming previous ADMM-based SNN pruning methods by over 8x in compression efficiency.
- Energy Efficiency: The resulting SNNs were 12.2x more efficient than parameter-equivalent ANNs and 38.7x more efficient than standard uncompressed ANNs.
- Latency Reduction: The hybrid training approach allowed models to reach peak accuracy within 100–150 time steps, a massive improvement over traditional conversion methods that often require thousands of steps.
Conclusion
Spike-Thrift demonstrates that algorithmic sparsity and attention-based distillation can overcome the inherent instabilities of SNN training. By reducing both the model size and the temporal activity (spikes), the framework paves the way for deploying high-performance deep learning models on energy-constrained neuromorphic hardware. Future work will explore alternative encoding schemes and validation on physical neuromorphic chips like Loihi.
Summary updated