Enhancing the Performance of Transformer-based Spiking Neural Networks by SNN-optimized Downsampling with Precise Gradient Backpropagation

Chenlin Zhou, Han Zhang, Zhaokun Zhou, Liutao Yu, Zhengyu Ma, Huihui Zhou, Xiaopeng Fan, Yonghong TianarXiv 2023 · DOI · arXiv

Introduction

Spiking Neural Networks (SNNs) are often hailed as the "third generation" of neural networks due to their biological plausibility and extreme energy efficiency on neuromorphic hardware. However, training deep SNNs—particularly those utilizing Transformer architectures like Spikformer—presents significant challenges. This paper identifies a critical bottleneck in current SNN designs: imprecise gradient backpropagation during downsampling.

The Problem: Imprecise Gradients

In traditional SNN architectures, a pooling layer (such as MaxPooling) is typically placed after the spiking Leaky Integrate-and-Fire (LIF) neuron. Because LIF neurons output discrete binary pulses (0 or 1), the subsequent MaxPooling operation loses the ability to distinguish between features of varying importance. If multiple neurons within a pooling window spike, the "max" value is simply 1, regardless of the underlying membrane potential. Consequently, during backpropagation, the gradient flows through an essentially arbitrary path. This "imprecision" hinders effective learning and limits the performance of advanced spiking transformers.

Methodology: The CML Module

To address this, the authors propose the ConvBN-MaxPooling-LIF (CML) module. The core innovation is a strategically simple reordering of the layer sequence:

  1. Traditional Approach: Convolution → Batch Norm → LIF (Spiking) → MaxPooling.
  2. CML Approach: Convolution → Batch Norm → MaxPooling → LIF (Spiking).

By moving MaxPooling before the LIF neuron, the pooling operation acts on high-precision, continuous floating-point values (membrane potentials) rather than binary spikes. This ensures that the gradient is mathematically guaranteed to flow through the element containing the most feature information.

Comparison of traditional vs. CML downsampling architectures and gradient flow.

Experimental Results

The CML module was evaluated across a comprehensive suite of benchmarks, including static image datasets (ImageNet-1K, CIFAR10/100) and neuromorphic datasets (CIFAR10-DVS, DVS128-Gesture).

Performance on ImageNet-1K

The integration of CML into the Spikingformer architecture yielded a new state-of-the-art for directly trained SNNs. As shown in the results, the CML-enhanced models consistently outperformed their baselines across various parameter scales.

Performance comparison on ImageNet-1K showing CML improvements across different model scales.

Key Findings

  • Accuracy Gains: On ImageNet-1K, Spikingformer + CML achieved 77.64% Top-1 accuracy, a +1.79% improvement over the baseline. Similar gains were observed on CIFAR100 (+1.16%) and neuromorphic datasets like CIFAR10-DVS (reaching 81.4%).
  • Efficiency: CML reduces computational overhead. By performing MaxPooling before the LIF neurons, the number of expensive spiking neuron simulations is reduced by 75% (assuming a stride of 2).
  • Gradient Fidelity: Theoretical proofs and empirical results confirm that preserving continuous values during spatial reduction is vital for maintaining gradient integrity in quantized systems like SNNs.

Conclusion

This research provides a vital architectural refinement for SNNs. By optimizing the order of operations in downsampling, the CML module resolves the long-standing issue of imprecise gradients in spiking transformers. While future work is needed to benchmark real-world power savings on specific neuromorphic hardware, CML represents a significant leap forward in making SNNs competitive with traditional artificial neural networks in both accuracy and efficiency.

Summary updated

Abstract

Deep spiking neural networks (SNNs) have drawn much attention in recent years because of their low power consumption, biological rationality and event-driven property. However, state-of-the-art deep SNNs (including Spikformer and Spikingformer) suffer from a critical challenge related to the imprecise gradient backpropagation. This problem arises from the improper design of downsampling modules in these networks, and greatly hampering the overall model performance. In this paper, we propose ConvBN-MaxPooling-LIF (CML), an SNN-optimized downsampling with precise gradient backpropagation. We prove that CML can effectively overcome the imprecision of gradient backpropagation from a theoretical perspective. In addition, we evaluate CML on ImageNet, CIFAR10, CIFAR100, CIFAR10-DVS, DVS128-Gesture datasets, and show state-of-the-art performance on all these datasets with significantly enhanced performances compared with Spikingformer. For instance, our model achieves 77.64 $\%$ on ImageNet, 96.04 $\%$ on CIFAR10, 81.4$\%$ on CIFAR10-DVS, with + 1.79$\%$ on ImageNet, +1.16$\%$ on CIFAR100 compared with Spikingformer.
BibTeX
@misc{zhou2023enhancing,
  author = {Chenlin Zhou and Han Zhang and Zhaokun Zhou and Liutao Yu and Zhengyu Ma and Huihui Zhou and Xiaopeng Fan and Yonghong Tian},
  title = {{Enhancing the Performance of Transformer-based Spiking Neural Networks by SNN-optimized Downsampling with Precise Gradient Backpropagation}},
  year = {2023},
  eprint = {2305.05954},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.05954},
}
Figure