Spike-Thrift: Towards Energy-Efficient Deep Spiking Neural Networks by Limiting Spiking Activity via Attention-Guided Compression

Souvik Kundu, Gourav Datta, Massoud Pedram, Peter A. Beerel

Abstract

Deep Spiking Neural Networks (SNNs) are promising for low-power neuromorphic computing but are hampered by high inference latency and training instabilities during compression. This paper introduces Spike-Thrift, a two-stage framework designed to optimize SNNs for both sparsity and energy efficiency. By employing Attention-Guided Compression (AGC) to stabilize pruning in networks lacking Batch Normalization and subsequent sparse-learning fine-tuning, Spike-Thrift achieves state-of-the-art compression ratios (up to 33.4x) and significant energy savings (up to 38.7x) over traditional artificial neural networks (ANNs).

Introduction

Spiking Neural Networks (SNNs) mimic the brain's event-driven communication using discrete binary spikes. While inherently energy-efficient on specialized hardware, SNNs suffer from two primary issues: the "Training Gap," where converting trained ANNs to SNNs requires excessive time steps, and the "Compression Paradox," where standard pruning techniques destabilize networks that lack Batch Normalization (BN).

Gradient magnitude distributions with and without Batch Normalization, highlighting the instability in BN-less networks.

As illustrated in the figure above, networks without BN exhibit highly concentrated gradient magnitudes, which can lead to gradient explosion or vanishing during pruning. Spike-Thrift addresses these challenges by using attention mechanisms to guide the compression process.

Methodology

The Spike-Thrift pipeline consists of two distinct stages aimed at structural and operational efficiency.

Stage 1: Attention-Guided Compression (AGC)

To prune models without the stabilizing effect of Batch Normalization, the authors utilize a "meta-model" (a dense teacher network). An attention-transfer loss term is introduced to ensure the sparse "student" model mimics the critical feature maps of the teacher. This attention-guided approach provides the necessary gradient stability to prune the network effectively before it is converted into the spiking domain.

Stage 2: Sparse-Learning SNN Training

Following compression, the model is converted into an SNN. To recover accuracy and minimize inference latency, the authors apply a sparse-learning (SL) strategy. This involves fine-tuning the SNN using surrogate gradients to handle the non-differentiable nature of spikes. By updating only the non-zero weights, the network learns to maintain high performance with significantly fewer time steps ($T$), directly reducing energy consumption.

Overview of the Spike-Thrift framework, showing (a) the attention-guided compression process and (b) the detailed spiking operation within the network.

Experimental Results

The framework was evaluated on CIFAR-10, CIFAR-100, and Tiny-ImageNet using VGG and ResNet architectures.

  • Compression and Accuracy: Spike-Thrift achieved a 33.4x compression ratio on VGG16 for CIFAR-10 with minimal accuracy loss, outperforming previous ADMM-based SNN pruning methods by over 8x in compression efficiency.
  • Energy Efficiency: The resulting SNNs were 12.2x more efficient than parameter-equivalent ANNs and 38.7x more efficient than standard uncompressed ANNs.
  • Latency Reduction: The hybrid training approach allowed models to reach peak accuracy within 100–150 time steps, a massive improvement over traditional conversion methods that often require thousands of steps.

Conclusion

Spike-Thrift demonstrates that algorithmic sparsity and attention-based distillation can overcome the inherent instabilities of SNN training. By reducing both the model size and the temporal activity (spikes), the framework paves the way for deploying high-performance deep learning models on energy-constrained neuromorphic hardware. Future work will explore alternative encoding schemes and validation on physical neuromorphic chips like Loihi.

Summary updated

Abstract

The increasing demand for on-chip edge intelligence has motivated the exploration of algorithmic techniques and specialized hardware to reduce the computation energy of current machine learning models. In particular, deep spiking neural networks (SNNs) have gained interest because their event-driven hardware implementations can consume very low energy. However, minimizing average spiking activity and thus energy consumption while preserving accuracy in deep SNNs remains a significant challenge and opportunity. This paper proposes a novel two-step SNN compression technique to reduce their spiking activity while maintaining accuracy that involves compressing specifically-designed artificial neural networks (ANNs) that are then converted into the target SNNs. Our approach uses an ultra-high ANN compression technique that is guided by the attention-maps of an uncompressed meta-model. We then evaluate the firing threshold of each ANN layer and start with the trained ANN weights to perform a sparse-learning-based supervised SNN training to minimize the number of time steps required while retaining compression. To evaluate the merits of the proposed approach, we performed experiments with variants of VGG and ResNet, on both CIFAR-10 and CIFAR-100, and VGG16 on Tiny-ImageNet. SNN models generated through the proposed technique yield state-of-the-art compression ratios of up to 33.4× with no significant drop in accuracy compared to baseline unpruned counterparts. As opposed to the existing SNN pruning methods we achieve up to 8.3× better compression with no drop in accuracy. Moreover, compressed SNN models generated by our methods can have up to 12.2× better compute energy-efficiency compared to ANNs that have a similar number of parameters.
BibTeX
@inproceedings{kundu2021spike,
  author = {Souvik Kundu and Gourav Datta and Massoud Pedram and Peter A. Beerel},
  title = {{Spike-Thrift: Towards Energy-Efficient Deep Spiking Neural Networks by Limiting Spiking Activity via Attention-Guided Compression}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00400},
}
Figure