Hessian Aware Quantization of Spiking Neural Networks

Hin Wai Lui, Emre Neftci

This paper presents a two-pronged approach to making Spiking Neural Networks (SNNs) more efficient for neuromorphic hardware: a simplified neuron model that slashes memory overhead and a Hessian-based quantization strategy that intelligently allocates bit-precision across network layers. By using the Hessian trace to identify which layers are most sensitive to noise, the authors achieved a 58% reduction in model size with a negligible 0.3% drop in accuracy.

TL;DR

The authors introduce a simplified Leaky Integrate-and-Fire (LIF) neuron model and a mixed-precision quantization scheme guided by Hessian trace analysis. This allows Spiking Neural Networks to be compressed by over 50% while maintaining near-peak accuracy, making them ideal for deployment on resource-constrained neuromorphic hardware.

What Problem Does This Paper Address?

Spiking Neural Networks (SNNs) are the "third generation" of neural networks, designed to mimic the brain's energy efficiency by processing information via discrete spikes. While they are theoretically perfect for neuromorphic chips (like Intel’s Loihi or IBM’s TrueNorth), practical deployment faces two major hurdles:

  1. Memory Overhead of Training Models: Popular SNN training frameworks (like DECOLLE) often use complex neuron models with multiple state variables to remain compatible with gradient-based learning. This consumes significant memory and compute power.
  2. The Quantization Dilemma: Neuromorphic hardware thrives on low-precision (e.g., 4-bit or 8-bit) arithmetic. However, simply "chopping off" bits across the whole network usually tanks accuracy. We need a way to determine which layers can handle low precision and which layers require higher precision to maintain performance.

Methods

The researchers attacked these problems with three key innovations:

  • Simplified Neuron Model: They modified the DECOLLE neuron model to reduce the number of state variables by 4-fold. Crucially, they proved this simplified model is mathematically equivalent to the standard LIF model during inference, meaning it only requires a single state variable (membrane potential) to run once trained.
  • Hessian Trace Analysis: The Hessian matrix represents the second-order derivative of the loss function. Its "trace" (the sum of its eigenvalues) serves as a proxy for sensitivity. A layer with a high Hessian trace is "sensitive"—small changes to its weights (like those caused by quantization) lead to large jumps in error. Conversely, a low trace indicates a "robust" layer.
  • Mixed Precision Quantization: Using the Hutchinson method for fast estimation, the authors calculated the Hessian trace for each layer. They then used these values to manually assign bit-precisions (e.g., giving 16 bits to sensitive layers and 8 or 4 bits to robust ones). They employed stochastic rounding during training to help the network adapt to these lower-precision weights.

Datasets

The authors evaluated their approach using the N-MNIST (Neuromorphic-MNIST) dataset. This is an event-based version of the classic MNIST handwritten digit dataset, recorded using a DVS (Dynamic Vision Sensor) camera. It consists of 60,000 training samples and 10,000 testing samples across 10 classes.

Key Insights and Ideas

  • Sensitivity Increases with Depth: The Hessian trace increased by four orders of magnitude from the first layer (L1) to the third layer (L3). This confirms that deeper layers in these SNNs are significantly more sensitive to weight perturbations than early layers.
  • The 16-bit "Free Lunch": Moving from full 32-bit floating point (FP32) to 16-bit fixed-point resulted in a 50% reduction in network size with zero loss in accuracy. This suggests that standard SNN training is significantly over-parameterized in terms of bit-depth.
  • Hessian-Guided Allocation Works: The experiments showed that assigning fewer bits to L1 (the least sensitive layer) had a much smaller impact on accuracy than reducing bits in L3. This validates the Hessian trace as a reliable metric for mixed-precision scheduling in spiking architectures.

Key Results

The researchers tested various bit-configurations on a 3-layer convolutional SNN:

  • Baseline (FP32): 98.1% accuracy, 4.84 MB size.
  • Uniform 16-bit: 98.1% accuracy, 2.42 MB size (50% reduction).
  • Optimal Mixed Precision (8, 8, 16 bits): 97.8% accuracy, 2.01 MB size (58% reduction).
  • Aggressive Quantization (4, 4, 4 bits): Accuracy dropped significantly to 81.1%, highlighting the necessity of the mixed-precision approach for maintaining high performance.

Limitations and Future Work

  • Manual Allocation: While the Hessian trace provided the data for bit-allocation, the authors still assigned the bits manually. Future work could automate this using a Pareto Frontier method to find the absolute optimal bit-depth for every single parameter.
  • Small Scale: The study used a relatively shallow 3-layer network. The authors note that SNNs generally struggle with depth in current benchmarks, and testing this on much deeper architectures remains a challenge.
  • Simulation vs. Hardware: The quantization was simulated in PyTorch (using FP32 underlying math). The next logical step is to deploy these models on actual neuromorphic hardware to measure real-world power savings and throughput gains.

Summary updated

Abstract

To achieve the low latency, high throughput, and energy efficiency benefits of Spiking Neural Networks (SNNs), reducing the memory and compute requirements when running on a neuromorphic hardware is an important step. Neuromorphic architecture allows massively parallel computation with variable and local bit-precisions. However, how different bit-precisions should be allocated to different layers or connections of the network is not trivial. In this work, we demonstrate how a layer-wise Hessian trace analysis can measure the sensitivity of the loss to any perturbation of the layer’s weights, and this can be used to guide the allocation of a layer-specific bit-precision when quantizing an SNN. In addition, current gradient based methods of SNN training use a complex neuron model with multiple state variables, which is not ideal for compute and memory efficiency. To address this challenge, we present a simplified neuron model that reduces the number of state variables by 4-fold while still being compatible with gradient based training. We find that the impact on model accuracy when using a layer-wise bit-precision correlated well with that layer’s Hessian trace. The accuracy of the optimal quantized network only dropped by 0.3%, yet the network size was reduced by 58%. This reduces memory usage and allows fixed-point arithmetic with simpler digital circuits to be used, increasing the overall throughput and energy efficiency.
BibTeX
@inproceedings{lui2021hessian,
  author = {Hin Wai Lui and Emre Neftci},
  title = {{Hessian Aware Quantization of Spiking Neural Networks}},
  booktitle = {ICONS},
  year = {2021},
  doi = {10.1145/3477145.3477158},
}

Loading related content...

Figure