Spiking-YOLO: Spiking Neural Network for Real-time Object Detection

Sei Joon Kim, Seongsik Park, Byunggook Na, Sungroh Yoon

TL;DR

Spiking-YOLO is the first deep spiking neural network (SNN) successfully adapted for object detection, achieving near-lossless performance compared to its deep neural network (DNN) counterpart. By introducing channel-wise normalization and signed neurons with imbalanced thresholds, the model retains 98% of Tiny YOLO’s accuracy while consuming up to 280 times less energy on neuromorphic hardware.

What Problem Does This Paper Address?

Spiking Neural Networks (SNNs) process information using discrete, sparse "spikes" over time, mimicking the energy-efficient nature of the human brain. While SNNs are highly promising for low-power edge devices, they are notoriously difficult to train from scratch due to their non-differentiable operations.

To bypass this, researchers typically convert pre-trained conventional Deep Neural Networks (DNNs) into SNNs. However, this conversion has historically only worked for simple image classification tasks on small datasets (like MNIST or CIFAR). When applied to complex regression tasks like object detection—which requires predicting precise floating-point coordinates for bounding boxes—converted SNNs fail completely.

The authors identify two primary culprits for this failure:

  1. Under-activation from layer-wise normalization: Standard conversion methods normalize weights layer by layer, which severely squashes the firing rates of channels with smaller activation ranges, destroying numerical precision.
  2. The absence of Leaky-ReLU support: Object detection models rely heavily on Leaky-ReLU activation functions to preserve negative values. Standard SNN neurons only handle positive thresholds, completely discarding negative activations (which account for over 51% of the features in Tiny YOLO).

Methods

The authors introduce two novel techniques to enable high-precision regression in deep SNNs:

1. Channel-wise Data-based Normalization (Channel-norm)

Instead of finding a single maximum activation value for an entire layer, the proposed Channel-norm normalizes weights and thresholds on a fine-grained, channel-by-channel basis using the 99.9th percentile of activations from the training set. This ensures that neurons in every channel maintain a healthy, balanced firing rate, preventing information loss in channels with naturally lower activation magnitudes.

Overview of the proposed channel-wise normalization method.

2. Signed Neuron with Imbalanced Threshold (IBT)

To accurately replicate Leaky-ReLU ($f(x) = x$ for $x \ge 0$, and $f(x) = \alpha x$ for $x < 0$) without introducing expensive floating-point multiplications on neuromorphic hardware, the authors designed a signed neuron with two distinct thresholds:

  • A positive threshold: $V_{th, pos} = V_{th}$
  • A negative threshold: $V_{th, neg} = -\frac{1}{\alpha}V_{th}$

If a neuron's membrane potential goes negative, it requires $\frac{1}{\alpha}$ times more integration to fire a negative spike ($-1$). This elegantly maps the continuous slope $\alpha$ into a discrete temporal firing frequency ratio.

Overview of the signed neuron featuring an imbalanced threshold.

Datasets

The model was evaluated on two standard, non-trivial object detection benchmarks:

  • PASCAL VOC 2007
  • MS COCO

Key Insights and Ideas

  • The Channel Variance Bottleneck: In deep convolutional layers, the variance of maximum activations across different channels is massive. As shown in Figure 1, layer-wise normalization forces many channels to have a normalized activation near 0, translating to an unacceptably low spike rate that cannot transmit precise regression values within a reasonable timeframe.
  • Higher Firing Rates = Faster Convergence: By moving to channel-wise normalization, the network eliminates extremely low firing rates. More neurons fire regularly and at higher rates (up to 80%), allowing the network to transmit accurate information much faster.
Figure 3: Firing rate distribution for layer-norm and channel- norm on channel 2 of Conv1 layer (Tiny YOLO)

Key Results

  • Near-Lossless Accuracy: Spiking-YOLO achieved 51.83% mAP on PASCAL VOC and 25.66% mAP on MS COCO, which is remarkably close to the original floating-point Tiny YOLO DNN performance (53.01% and 26.24% mAP, respectively). Without the proposed methods, the converted SNNs failed entirely, scoring near 0% mAP.
  • Massive Energy Savings: Based on energy consumption metrics for neuromorphic chips (simulated TrueNorth architecture), Spiking-YOLO consumes 280 times less energy than the baseline Tiny YOLO running on a Titan V100 GPU.
  • Faster Temporal Convergence: Thanks to Channel-norm, Spiking-YOLO reaches its peak accuracy 2.3$\times$ to 4$\times$ faster (requiring fewer time steps, around 3,500) compared to standard layer-wise normalization methods.

Limitations and Future Work

  • Architecture Constraints: The evaluation is restricted to Tiny YOLO, which is a lightweight, simplified variant of YOLO. Scaling this conversion method to larger, more complex architectures like YOLOv3 or YOLOv4 remains an open challenge.
  • Latency Trade-off: Although energy consumption is incredibly low, the SNN still requires thousands of time steps (3,500 to 8,000) to reach maximum precision. This temporal decoding scheme introduces an inherent latency penalty that might challenge ultra-stringent real-time constraints.

Summary updated

Abstract

Over the past decade, deep neural networks (DNNs) have demonstrated remarkable performance in a variety of applications. As we try to solve more advanced problems, increasing demands for computing and power resources has become inevitable. Spiking neural networks (SNNs) have attracted widespread interest as the third-generation of neural networks due to their event-driven and low-powered nature. SNNs, however, are difficult to train, mainly owing to their complex dynamics of neurons and non-differentiable spike operations. Furthermore, their applications have been limited to relatively simple tasks such as image classification. In this study, we investigate the performance degradation of SNNs in a more challenging regression problem (i.e., object detection). Through our in-depth analysis, we introduce two novel methods: channel-wise normalization and signed neuron with imbalanced threshold, both of which provide fast and accurate information transmission for deep SNNs. Consequently, we present a first spiked-based object detection model, called Spiking-YOLO. Our experiments show that Spiking-YOLO achieves remarkable results that are comparable (up to 98%) to those of Tiny YOLO on non-trivial datasets, PASCAL VOC and MS COCO. Furthermore, Spiking-YOLO on a neuromorphic chip consumes approximately 280 times less energy than Tiny YOLO and converges 2.3 to 4 times faster than previous SNN conversion methods.
BibTeX
@inproceedings{kim2020spiking,
  author = {Sei Joon Kim and Seongsik Park and Byunggook Na and Sungroh Yoon},
  title = {{Spiking-YOLO: Spiking Neural Network for Real-time Object Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6787},
}
Figure