Introduction
Spiking Neural Networks (SNNs) are often hailed as the "third generation" of neural networks due to their biological plausibility and extreme energy efficiency on neuromorphic hardware. However, training deep SNNs—particularly those utilizing Transformer architectures like Spikformer—presents significant challenges. This paper identifies a critical bottleneck in current SNN designs: imprecise gradient backpropagation during downsampling.
The Problem: Imprecise Gradients
In traditional SNN architectures, a pooling layer (such as MaxPooling) is typically placed after the spiking Leaky Integrate-and-Fire (LIF) neuron. Because LIF neurons output discrete binary pulses (0 or 1), the subsequent MaxPooling operation loses the ability to distinguish between features of varying importance. If multiple neurons within a pooling window spike, the "max" value is simply 1, regardless of the underlying membrane potential. Consequently, during backpropagation, the gradient flows through an essentially arbitrary path. This "imprecision" hinders effective learning and limits the performance of advanced spiking transformers.
Methodology: The CML Module
To address this, the authors propose the ConvBN-MaxPooling-LIF (CML) module. The core innovation is a strategically simple reordering of the layer sequence:
- Traditional Approach: Convolution → Batch Norm → LIF (Spiking) → MaxPooling.
- CML Approach: Convolution → Batch Norm → MaxPooling → LIF (Spiking).
By moving MaxPooling before the LIF neuron, the pooling operation acts on high-precision, continuous floating-point values (membrane potentials) rather than binary spikes. This ensures that the gradient is mathematically guaranteed to flow through the element containing the most feature information.
Experimental Results
The CML module was evaluated across a comprehensive suite of benchmarks, including static image datasets (ImageNet-1K, CIFAR10/100) and neuromorphic datasets (CIFAR10-DVS, DVS128-Gesture).
Performance on ImageNet-1K
The integration of CML into the Spikingformer architecture yielded a new state-of-the-art for directly trained SNNs. As shown in the results, the CML-enhanced models consistently outperformed their baselines across various parameter scales.
Key Findings
- Accuracy Gains: On ImageNet-1K, Spikingformer + CML achieved 77.64% Top-1 accuracy, a +1.79% improvement over the baseline. Similar gains were observed on CIFAR100 (+1.16%) and neuromorphic datasets like CIFAR10-DVS (reaching 81.4%).
- Efficiency: CML reduces computational overhead. By performing MaxPooling before the LIF neurons, the number of expensive spiking neuron simulations is reduced by 75% (assuming a stride of 2).
- Gradient Fidelity: Theoretical proofs and empirical results confirm that preserving continuous values during spatial reduction is vital for maintaining gradient integrity in quantized systems like SNNs.
Conclusion
This research provides a vital architectural refinement for SNNs. By optimizing the order of operations in downsampling, the CML module resolves the long-standing issue of imprecise gradients in spiking transformers. While future work is needed to benchmark real-world power savings on specific neuromorphic hardware, CML represents a significant leap forward in making SNNs competitive with traditional artificial neural networks in both accuracy and efficiency.
Summary updated