Abstract
To achieve the low latency, high throughput, and energy efficiency benefits of Spiking Neural Networks (SNNs), reducing the memory and compute requirements when running on a neuromorphic hardware is an important step. Neuromorphic architecture allows massively parallel computation with variable and local bit-precisions. However, how different bit-precisions should be allocated to different layers or connections of the network is not trivial. In this work, we demonstrate how a layer-wise Hessian trace analysis can measure the sensitivity of the loss to any perturbation of the layer's weights, and this can be used to guide the allocation of a layer-specific bit-precision when quantizing an SNN. In addition, current gradient based methods of SNN training use a complex neuron model with multiple state variables, which is not ideal for compute and memory efficiency. To address this challenge, we present a simplified neuron model that reduces the number of state variables by 4-fold while still being compatible with gradient based training. We find that the impact on model accuracy when using a layer-wise bit-precision correlated well with that layer's Hessian trace. The accuracy of the optimal quantized network only dropped by 0.2%, yet the network size was reduced by 58%. This reduces memory usage and allows fixed-point arithmetic with simpler digital circuits to be used, increasing the overall throughput and energy efficiency.
Introduction
Spiking Neural Networks (SNNs) are often hailed as the "third generation" of neural networks, promising brain-like energy efficiency by processing information via discrete spikes. While theoretically ideal for neuromorphic hardware like Intel’s Loihi, practical deployment is hindered by high memory overhead during training and the difficulty of effective quantization. Standard SNN training frameworks often rely on complex neuron models with multiple state variables, and uniform quantization frequently leads to significant accuracy degradation. This paper addresses these challenges through a simplified neuron model and a Hessian-guided mixed-precision quantization strategy.
Methodology
Simplified Neuron Model
To improve memory and compute efficiency, the researchers modified the DECOLLE neuron model, reducing the number of state variables by four-fold. This simplified model remains mathematically equivalent to the standard Leaky Integrate-and-Fire (LIF) model during inference, requiring only a single state variable (membrane potential) once trained.
Hessian-Guided Quantization
The core of the quantization strategy lies in Hessian trace analysis. The Hessian matrix represents the second-order derivative of the loss function, and its trace (the sum of its eigenvalues) serves as a proxy for a layer's sensitivity to weight perturbations. Layers with a high Hessian trace are considered "sensitive," meaning small changes—such as those introduced by quantization—can lead to large errors. Conversely, layers with a low trace are considered "robust."
By estimating the Hessian trace using the Hutchinson method, the authors can intelligently allocate bit-precisions across the network. They employed stochastic rounding during training to help the network adapt to these lower-precision weights.
Experimental Results
The approach was evaluated on the N-MNIST (Neuromorphic-MNIST) dataset, an event-based version of the classic MNIST dataset recorded with a Dynamic Vision Sensor (DVS).
Sensitivity and Bit Allocation
Analysis revealed that sensitivity increases significantly with network depth. The Hessian trace for the third layer (L3) was found to be orders of magnitude larger than that of the first layer (L1). This confirms that deeper layers require higher precision to maintain performance, while early layers can tolerate more aggressive quantization.
Quantization Performance
The researchers tested various bit-configurations on a 3-layer convolutional SNN. Transitioning from 32-bit floating point (FP32) to uniform 16-bit fixed-point achieved a 50% reduction in size with no loss in accuracy. By applying Hessian-guided mixed precision (e.g., 8 bits for L1 and L2, 16 bits for L3), the model size was reduced by 58% while maintaining a high accuracy of 97.8%.
Conclusion
This work demonstrates that Hessian trace analysis is a reliable metric for scheduling mixed-precision quantization in SNNs. Combined with a simplified neuron model, this approach significantly reduces the memory and computational footprint of spiking architectures without sacrificing accuracy. Future work will focus on automating bit-allocation and deploying these optimized models on physical neuromorphic hardware to validate real-world efficiency gains.
Summary updated