The Hardware Impact of Quantization and Pruning for Weights in Spiking Neural Networks

Clemens Js Schaefer, Pooria Taheri, Mark Horeni, Siddharth Joshi

TL;DR

This paper investigates how common model compression techniques like quantization and pruning affect the energy efficiency and accuracy of Spiking Neural Networks (SNNs) on hardware. They find that aggressive quantization, even down to ternary weights, can maintain accuracy while significantly reducing energy, often outperforming pruning alone.

What Problem Does This Paper Address?

Spiking Neural Networks (SNNs) hold immense promise for energy-efficient and low-latency AI, mimicking the brain's computational style. However, deploying these networks on real-world hardware requires efficient implementations. Two key techniques for this are quantization (reducing the precision of weights, e.g., from 32-bit floats to fewer bits) and pruning (removing less important weights to reduce model size and computation). While these techniques are well-studied for traditional Artificial Neural Networks (ANNs), their combined impact and interaction on SNN hardware accelerators, especially concerning energy efficiency, remain largely unknown. This paper aims to bridge this gap by systematically studying how different combinations of quantization and pruning affect SNN performance on a representative hardware architecture.

Methods

The authors employ a state-of-the-art SNN architecture for gesture recognition on dynamic vision sensor (DVS) data. Their methodology involves:

  • Quantization: They explore various bit-widths for weights, including fixed-point integer quantization and ternary weights (using only three distinct values). They utilize quantization-aware training to minimize accuracy loss.
  • Pruning: Global unstructured pruning is applied, where weights are ranked by magnitude and a certain percentage are zeroed out. They investigate sparsity targets up to 97.5%.
  • Combined Strategies: They study three approaches:
  • Quantization Only: Applying quantization to the weights.
  • Pruning Only: Applying pruning to the weights.
  • Cumulative: First finetuning with pruning, then applying quantization-aware training.
  • Jointly: Simultaneously pruning and quantizing the weights during training.
  • Hardware Evaluation: The SNN model is evaluated on a simulated Eyeriss-like spiking hardware architecture. This architecture is designed for energy efficiency and incorporates features like shared activation buffers and digital cores with scratchpad memories. Energy consumption is estimated using high-level component-based tools.
  • Sparsity Encoding: For pruned models, they explore different sparse storage formats like uncompressed bitmask (UBM), uncompressed offset pair (UOP), coordinate payload (CP), and run-length encoding (RLE) to represent sparse weights efficiently.

Datasets

  • DVS Gesture Dataset: This dataset is used for training and evaluating the SNN model for gesture recognition. It's an event-based dataset, meaning it captures data only when there's a change in pixel intensity, making it suitable for SNNs.

Key Insights and Ideas

  • Ternary Quantization is Surprisingly Effective: The paper demonstrates that aggressive quantization down to ternary weights can be applied without significant accuracy loss on their chosen SNN model and task. This is a key finding because it suggests that SNNs can tolerate very low precision.
  • Quantization's Synergy with Sparsity: While pruning aims to reduce redundancy by zeroing weights, quantization can induce sparsity (many weights become zero or a few specific values after quantization). Ternary quantization, in particular, creates a high degree of induced sparsity, which can be leveraged by hardware for significant energy savings.
  • Pruning Alone Has Limits: Pruning alone can achieve iso-accuracy (same accuracy as the baseline) up to about 80% sparsity. However, beyond this point, accuracy degrades, and the energy savings are less pronounced compared to aggressive quantization.
  • Combined Approaches Offer Pareto Optimality: When accuracy degradation is acceptable, hybrid pruning and quantization schemes can achieve a favorable trade-off on the energy-accuracy Pareto frontier, allowing for models tailored to specific SWAP (Size, Weight, Area, Power) constraints.
  • Hardware Matters: The energy breakdown highlights that data movement, especially to and from DRAM, is a significant energy consumer. Ternarization, by reducing weight precision and inducing sparsity, can lead to more efficient data handling and computation on the target hardware.

Key Results

  • Iso-Accuracy Performance:
    • Ternary quantization (3-bit) achieved iso-accuracy with a significant energy reduction compared to the baseline 8-bit model.
  • Pruning alone maintained iso-accuracy up to 80% sparsity but resulted in 45% more energy than the best quantization strategy.
  • Energy Breakdown: Ternarization incurs minimal DRAM access, leading to substantial efficiency gains. For 8-bit sparse models, storing metadata for sparsity incurs significant costs.
  • Sparsity Encoding Impact: For models with low sparsity, UOP performs best. However, for models with considerable sparsity (like pruned or highly quantized ones), RLE outperforms other formats, offering a slight advantage in energy and EDP.
  • Combined Strategies: Jointly pruned and quantized models can achieve better energy efficiency than either technique alone when some accuracy loss is tolerated, forming a Pareto curve.

Limitations and Future Work

  • Model and Task Specificity: The findings are specific to the chosen SNN architecture and the DVS gesture recognition task. The authors acknowledge that the impact of quantization and pruning might differ for other SNN architectures or tasks, especially those highly dependent on precise spike timing.
  • Hardware Simulation: The energy evaluation is based on a simulated Eyeriss-like architecture. Real-world hardware implementation might reveal different nuances.
  • Future Work: The authors suggest further exploring hybrid pruning and quantization schemes to achieve a wider range of SWAP trade-offs. Investigating the impact on tasks more sensitive to spike timing would also be a valuable direction.

Summary updated

Abstract

Energy efficient implementations and deployments of Spiking neural networks (SNNs) have been of great interest due to the possibility of developing artificial systems that can achieve the computational powers and energy efficiency of the biological brain. Efficient implementations of SNNs on modern digital hardware are also inspired by advances in machine learning and deep neural networks (DNNs). Two techniques widely employed in the efficient deployment of DNNs -- the quantization and pruning of parameters, can both compress the model size, reduce memory footprints, and facilitate low-latency execution. The interaction between quantization and pruning and how they might impact model performance on SNN accelerators is currently unknown. We study various combinations of pruning and quantization in isolation, cumulatively, and simultaneously (jointly) to a state-of-the-art SNN targeting gesture recognition for dynamic vision sensor cameras (DVS). We show that this state-of-the-art model is amenable to aggressive parameter quantization, not suffering from any loss in accuracy down to ternary weights. However, pruning only maintains iso-accuracy up to 80% sparsity, which results in 45% more energy than the best quantization on our architectural model. Applying both pruning and quantization can result in an accuracy loss to offer a favourable trade-off on the energy-accuracy Pareto-frontier for the given hardware configuration.
BibTeX
@article{schaefer2023hardware,
  author = {Clemens JS Schaefer and Pooria Taheri and Mark Horeni and Siddharth Joshi},
  title = {{The Hardware Impact of Quantization and Pruning for Weights in Spiking Neural Networks}},
  journal = {IEEE Trans. Circuits Syst. ii Express Briefs},
  year = {2023},
  doi = {10.1109/tcsii.2023.3260701},
}
Figure