Introduction
Spiking Neural Networks (SNNs) represent a bio-inspired paradigm of "event-driven" deep learning, offering significant energy efficiency advantages for deployment on specialized neuromorphic hardware. To utilize these models on resource-constrained edge devices, quantization—the process of converting high-precision floating-point numbers into low-bit integers—is essential. While Quantization-Aware Training (QAT) for weights is well-established, the quantization of a neuron's internal state, specifically its membrane potential, remains a critical challenge.
The Challenge of State Quantization
In recurrent SNNs, the internal state of a neuron evolves over time. Standard practices often involve training with high-precision states but deploying on hardware that utilizes fixed-precision integers. This discrepancy leads to accumulated "truncation errors," resulting in a significant performance gap between simulation and real-world deployment. SQUAT (Stateful Quantization-Aware Training) addresses this by integrating state quantization directly into the training loop.
The SQUAT Methodology
The SQUAT framework introduces two primary quantization schemes to manage membrane potentials:
- Uniform SQUAT: This approach divides the range of membrane potentials into equal intervals, similar to standard weight quantization.
- Exponential (Threshold-Centered) SQUAT: This novel contribution allocates more quantization levels near the neuron's firing threshold. The rationale is that a neuron's sensitivity is highest when it is close to the "criticality" of firing a spike.
To maintain differentiability during backpropagation, SQUAT employs a Straight-Through Estimator (STE), allowing gradients to flow through the non-differentiable quantization step as if it were a continuous function.
Experimental Evaluation
The efficacy of SQUAT was evaluated across three diverse datasets to ensure robustness across modalities:
- FashionMNIST: A standard image classification benchmark.
- Spiking Heidelberg Digits (SHD): An auditory dataset consisting of spoken digits.
- DVS Gesture Dataset: Complex temporal patterns captured with event-based neuromorphic cameras.
Key Results and Insights
The results demonstrate that SQUAT is particularly effective in ultra-low-bit regimes (2-bit and 4-bit) where standard Post-Training Quantization (PTQ) often fails.
- Salvaging Low-Bit Performance: On FashionMNIST, a 2-bit model using PTQ achieved only 16.81% accuracy. Integrating weight QAT with exponential SQUAT boosted this accuracy to 67.82%.
- Superiority of Exponential Quantization: Exponential quantization consistently outperformed uniform quantization. For instance, on the SHD dataset at 2-bit precision, exponential levels outperformed uniform levels by approximately 8%.
- Sensitivity to Scale: Larger and more complex datasets like DVS Gesture showed the highest sensitivity to state quantization, indicating that SQUAT will be vital for scaling SNNs to real-world applications.
Conclusion
SQUAT provides a robust method for bridging the gap between high-precision SNN simulations and low-precision hardware deployment. By focusing precision near the firing threshold, it enables the use of ultra-low-bit widths without catastrophic loss in accuracy. The integration of these methods into the snnTorch library ensures that SQUAT is accessible for broader research and development in neuromorphic computing.
Summary updated