Training Full Spike Neural Networks via Auxiliary Accumulation Pathway

Guangyao Chen, Peixi Peng, Guoqi Li, Yonghong TianarXiv 2023 · DOI · arXiv

Introduction

Spiking Neural Networks (SNNs) represent a promising paradigm for energy-efficient artificial intelligence, drawing inspiration from the sparse, event-driven communication of biological brains. By utilizing binary "spikes" (0 or 1), SNNs allow neuromorphic hardware to replace energy-intensive Multiply-Accumulate (MAC) operations with simpler, low-power Accumulation (AC) operations. However, training deep, high-performance SNNs while maintaining a "full-spike" (purely binary) signal remains a significant challenge.

The Challenge of Deep Full-Spike Networks

Traditional deep learning architectures, such as ResNets, rely on identity mappings via residual connections to facilitate gradient flow. In standard SNN implementations, these additions often result in multi-bit integers (e.g., 0, 1, 2...), which necessitates the use of MAC operations and negates the energy efficiency of the spiking model. Furthermore, binary signals are prone to information loss and vanishing gradients during backpropagation, as the derivative of the spiking function is often zero. Consequently, most state-of-the-art SNNs are "mixed-precision" rather than "full-spike," sacrificing hardware efficiency for training stability.

Dual-Stream Training (DST) Methodology

To address these limitations, researchers have introduced Dual-Stream Training (DST). This framework enables the training of deep Full-Spike Neural Networks (FSNNs) by utilizing two parallel pathways during the training phase:

  1. Full-Spike Propagation Pathway: This core pathway maintains binary signals throughout the network. To support identity mapping without increasing bit-width, standard arithmetic addition is replaced with element-wise logical functions (e.g., AND, OR, XOR, IAND). This ensures the output remains a binary spike train.
  2. Auxiliary Accumulation Pathway (AAP): This is a detachable stream that runs alongside the spike pathway. It accumulates spikes into floating-point values during training to compensate for information loss in the forward pass. Crucially, it provides a stable gradient (equal to 1) in the backward pass, effectively mitigating the vanishing gradient problem.
Architectural comparison between standard Spiking ResNets/Transformers and the proposed DST framework with Auxiliary Accumulation Pathways.

A key advantage of DST is that the AAP is "detachable." Once training is complete, the AAP can be removed, leaving a pure FSNN that is fully compatible with accumulation-only neuromorphic hardware. Alternatively, if the target hardware supports limited MAC operations, the AAP can be retained to further enhance accuracy.

Experimental Evaluation

The DST method was evaluated across diverse benchmarks, including static image datasets (ImageNet, CIFAR100) and neuromorphic event-based datasets (DVS Gesture, CIFAR10-DVS).

Key Results

  • Scalability and Depth: DST successfully enabled the training of FSNNs up to 101 layers deep. On ImageNet, an FSNN-101 achieved 68.38% Top-1 accuracy with only 4 time steps, outperforming previous full-spike models.
  • Energy Efficiency: On the DVS Gesture dataset, the proposed FSNN-7 achieved 97.57% accuracy while consuming only 1/10th of the energy required by previous state-of-the-art models like SEW ResNet. This highlights the power of maintaining pure AC operations.
  • Logical Operator Versatility: The study found that the optimal logical operator for residuals varies by task; for instance, IAND showed superior performance on neuromorphic datasets, while OR was more effective for certain static image tasks.

Conclusion

Dual-Stream Training provides a robust solution for training deep, high-performance FSNNs without compromising the energy benefits of binary spiking signals. By decoupling the training stability (via the AAP) from the inference efficiency (via logical residuals), DST paves the way for the deployment of sophisticated AI models on low-power neuromorphic hardware. Future work will focus on optimizing downsampling blocks and validating these models on physical neuromorphic chips.

Summary updated

Abstract

Due to the binary spike signals making converting the traditional high-power multiply-accumulation (MAC) into a low-power accumulation (AC) available, the brain-inspired Spiking Neural Networks (SNNs) are gaining more and more attention. However, the binary spike propagation of the Full-Spike Neural Networks (FSNN) with limited time steps is prone to significant information loss. To improve performance, several state-of-the-art SNN models trained from scratch inevitably bring many non-spike operations. The non-spike operations cause additional computational consumption and may not be deployed on some neuromorphic hardware where only spike operation is allowed. To train a large-scale FSNN with high performance, this paper proposes a novel Dual-Stream Training (DST) method which adds a detachable Auxiliary Accumulation Pathway (AAP) to the full spiking residual networks. The accumulation in AAP could compensate for the information loss during the forward and backward of full spike propagation, and facilitate the training of the FSNN. In the test phase, the AAP could be removed and only the FSNN remained. This not only keeps the lower energy consumption but also makes our model easy to deploy. Moreover, for some cases where the non-spike operations are available, the APP could also be retained in test inference and improve feature discrimination by introducing a little non-spike consumption. Extensive experiments on ImageNet, DVS Gesture, and CIFAR10-DVS datasets demonstrate the effectiveness of DST.
BibTeX
@misc{chen2023training,
  author = {Guangyao Chen and Peixi Peng and Guoqi Li and Yonghong Tian},
  title = {{Training Full Spike Neural Networks via Auxiliary Accumulation Pathway}},
  year = {2023},
  eprint = {2301.11929},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.11929},
}
Figure