Direct training high-performance spiking neural networks for object recognition and detection

Hong Zhang, Yang Li, Bin He, Xiongfei Fan, Yue Wang, Yu ZhangFrontiers in Neuroscience 2023 · DOI

Abstract

The SG ResNet and ASD modules are evaluated on multiple object recognition datasets, including the static ImageNet, CIFAR-100, CIFAR-10, and neuromorphic DVS-CIFAR10 datasets. Superior accuracy is demonstrated with a tiny simulation time step of four, specifically 94.52% top-1 accuracy on CIFAR-10 and 75.64% top-1 accuracy on CIFAR-100. Spiking RetinaNet is proposed using SG ResNet as the backbone and ASD module for information decoding as the first direct-training hybrid SNN-ANN detector for RGB images. Spiking RetinaNet with a SG ResNet34 backbone achieves an mAP of 0.296 on the object detection dataset MSCOCO.

Introduction

Spiking Neural Networks (SNNs) represent a biologically inspired paradigm for artificial intelligence, utilizing discrete binary pulses (spikes) to communicate information. This sparsity allows for extreme energy efficiency on neuromorphic hardware. However, scaling SNNs to the depth and performance levels of standard Artificial Neural Networks (ANNs) has been hindered by two primary challenges: the "Depth Problem" and the "Decoding Problem."

In traditional ANNs, Residual Networks (ResNets) utilize identity mappings to enable the training of very deep architectures. In the spiking domain, the addition of binary spikes to analog shortcuts often disrupts the spiking nature of the network or causes gradient instability. Furthermore, traditional SNN decoding relies on simple rate coding—counting spikes over time—which neglects the rich spatial and temporal information inherent in spike trains.

Methodology

Spiking Gate (SG) ResNet

To address the challenges of training deep SNNs, this research introduces the Spiking Gate (SG) ResNet. The core innovation is the Binary Selection Gate (BSG), which replaces the standard addition operation in residual blocks.

Comparison of residual block architectures: (A) Basic ANN block, (B) standard Spiking block, and (C) the proposed Spiking Gate block.

The BSG acts as a learnable switch that chooses between the identity mapping (input $X$) and the output of the spiking layers. This mechanism ensures the network remains "fully spiking" while allowing gradients to flow unimpeded through deep architectures, effectively solving the vanishing gradient problem.

Illustration of the gate mechanism (BSG) used to enable identity mapping in SNNs.

Attention Spike Decoder (ASD)

To improve information extraction from spike trains, the authors developed the Attention Spike Decoder (ASD). Unlike rate coding, ASD applies a multi-dimensional attention mechanism across three axes:

  • Temporal: Identifying critical time steps.
  • Channel: Weighting relevant feature maps.
  • Spatial: Focusing on informative regions within the image.

Spiking RetinaNet

By integrating SG ResNet as a backbone and ASD for decoding, the authors constructed Spiking RetinaNet, the first direct-training hybrid SNN-ANN detector designed for complex RGB image datasets like MSCOCO.

Experimental Results

The proposed methods were evaluated across several benchmarks, demonstrating state-of-the-art performance with minimal latency (simulation time step $T=4$).

Image Classification

On the CIFAR-10 and CIFAR-100 datasets, the models achieved top-1 accuracies of 94.52% and 75.64%, respectively. On the large-scale ImageNet dataset, SG ResNet50 reached 66.25% accuracy, surpassing previous direct-training SNN methodologies.

Object Detection

Spiking RetinaNet achieved an mAP of 0.296 on the MSCOCO dataset. While this is slightly below the performance of its pure ANN counterpart (0.319), the SNN version operates with significantly higher energy efficiency.

Energy Efficiency

The "neuromorphic dividend" is substantial: SG ResNet18 consumed 41.7 times less energy than a comparable ANN by replacing energy-intensive multiply-accumulate (MAC) operations with simple additions (ACs).

Conclusion

This work demonstrates that SNNs can achieve high performance and depth through gated identity mappings and attention-based decoding. While challenges remain—such as overfitting in larger models and the continued reliance on hybrid components for detection heads—the SG ResNet and ASD modules provide a robust framework for efficient, high-performance neuromorphic computing.

Summary updated

Abstract

The SG ResNet and ASD modules are evaluated on multiple object recognition datasets, including the static ImageNet, CIFAR-100, CIFAR-10, and neuromorphic DVS-CIFAR10 datasets. Superior accuracy is demonstrated with a tiny simulation time step of four, specifically 94.52% top-1 accuracy on CIFAR-10 and 75.64% top-1 accuracy on CIFAR-100. Spiking RetinaNet is proposed using SG ResNet as the backbone and ASD module for information decoding as the first direct-training hybrid SNN-ANN detector for RGB images. Spiking RetinaNet with a SG ResNet34 backbone achieves an mAP of 0.296 on the object detection dataset MSCOCO.
BibTeX
@inproceedings{zhang2023direct,
  author = {Hong Zhang and Yang Li and Bin He and Xiongfei Fan and Yue Wang and Yu Zhang},
  title = {{Direct training high-performance spiking neural networks for object recognition and detection}},
  booktitle = {Frontiers in Neuroscience},
  year = {2023},
  doi = {10.3389/fnins.2023.1229951},
}
Figure