Spiking CenterNet: A Distillation-boosted Spiking Neural Network for Object Detection

Lennard Bodden, Bach Ha, Franziska Schwaiger, Lars Kreuzberg, Sven BehnkeSubmission to ICLR 2024 · Rejected

Abstract

The demand for energy-efficient, real-time perception in autonomous systems has led to increased interest in Spiking Neural Networks (SNNs). However, training SNNs for complex tasks like object detection remains challenging due to their non-differentiable nature and the sparsity of information. This paper presents Spiking CenterNet, a novel SNN architecture that performs high-efficiency object detection on event-based camera data. By combining an anchor-free detection head with a knowledge distillation (KD) framework, the model achieves a state-of-the-art 0.229 mAP on the Prophesee GEN1 dataset while consuming significantly less energy than previous spiking detectors.

Introduction

Traditional Artificial Neural Networks (ANNs) are computationally expensive, relying on continuous, high-precision floating-point operations that are ill-suited for power-constrained edge AI applications. In contrast, SNNs process information via discrete "spikes," enabling massive energy savings on specialized neuromorphic hardware. Despite their potential, SNNs often struggle with accuracy in complex vision tasks. Most existing spiking detectors rely on complex post-processing like Non-Maximum Suppression (NMS) or inefficient ANN-to-SNN conversion methods. Spiking CenterNet addresses these limitations by adopting an inherently efficient, NMS-free architecture boosted by teacher-student learning.

Spiking CenterNet Architecture

The proposed model is a fully spiking implementation of CenterNet, an anchor-free detector that identifies objects as single points (centroids) rather than fitting numerous pre-defined bounding boxes.

Overview of the Spiking CenterNet architecture featuring a ResNet-18 encoder, M2U-Net decoder, and knowledge distillation from a teacher model.

Design Components:

  • Encoder-Decoder Backbone: The system utilizes a ResNet-18 encoder paired with a decoder inspired by M2U-Net. This configuration employs efficient upsampling and binary skip connections, which are critical for maintaining gradient flow in sparse spiking environments.
  • Parametric LIF Neurons: Standard ReLU activations are replaced with Parametric Leaky Integrate-and-Fire (PLIF) neurons. These neurons include learnable parameters that allow the network to better capture the complex temporal dynamics inherent in event-based data.
  • NMS-Free Detection: By leveraging CenterNet’s heatmap-based approach to locate local maxima, the model identifies objects directly. This eliminates the need for computationally expensive NMS steps common in other detectors like YOLO.

Knowledge Distillation Strategy

A key innovation of this work is the application of Knowledge Distillation (KD) to spiking object detection. The training process involves two stages: first, a standard, non-spiking ANN version of the model is trained to high accuracy; then, the SNN "student" is trained to match the output heatmaps of the ANN "teacher."

This distillation helps the SNN learn nuanced features that are typically lost in binary spike trains, effectively bridging the accuracy gap between noisy, sparse SNNs and high-precision ANNs.

Evaluation and Results

The model was evaluated on the Prophesee GEN1 Automotive Detection Dataset, which contains 39 hours of event-based recordings and 255,000 manual annotations.

Qualitative comparison between ground truth annotations and Spiking CenterNet predictions on the GEN1 dataset.

Performance Metrics:

  • Accuracy: Spiking CenterNet achieved a 0.229 mAP, a significant improvement over the previous state-of-the-art fully-spiking model (0.189 mAP).
  • Efficiency: The model demonstrates remarkable energy efficiency, consuming between 0.62 mJ and 1.0 mJ per time step. This is less than half the energy consumption of previous leading spiking detectors.
  • Sparsity: The network maintains high operational sparsity, with firing rates between 10.8% and 17.4%, ensuring that the majority of the hardware remains idle and power-efficient.

Conclusion

Spiking CenterNet represents a significant step forward in neuromorphic vision. By integrating anchor-free detection with knowledge distillation, the authors have created a model that is both accurate and exceptionally energy-efficient. While a performance gap still exists between SNNs and their ANN counterparts, this work provides a robust framework for future research in low-power, real-time object detection for robotics and autonomous vehicles.

Summary updated

Abstract

In the era of AI at the edge, self-driving cars, and climate change, the need for energy-efficient, small, embedded AI is growing. Spiking Neural Networks (SNNs) are a promising approach to address this challenge, with their event-driven information flow and sparse activations. We propose Spiking CenterNet for object detection on event data. It combines an SNN CenterNet adaptation with an efficient M2U-Net-based decoder. Our model significantly outperforms comparable previous work on Prophesee's challenging GEN1 Automotive Detection Dataset while using less than half the energy. Distilling the knowledge of a non-spiking teacher into our SNN further increases performance. To the best of our knowledge, our work is the first approach that takes advantage of knowledge distillation in the field of spiking object detection.
BibTeX
@misc{bodden2024spiking,
  author = {Lennard Bodden and Bach Ha and Franziska Schwaiger and Lars Kreuzberg and Sven Behnke},
  title = {{Spiking CenterNet: A Distillation-boosted Spiking Neural Network for Object Detection}},
  year = {2024},
  eprint = {2402.01287},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.01287},
}
Figure