TL;DR
This paper introduces BN-SNN, a novel Spiking Neural Network (SNN) architecture for object detection that leverages bistable integrate-and-fire (BIF) neurons within a ResNet backbone and an SSD detection head. BN-SNN significantly reduces inference latency and improves mean average precision (mAP) compared to existing SNN conversion methods, paving the way for more efficient and accurate real-time object detection.
What Problem Does This Paper Address?
Deep neural networks (DNNs), particularly Convolutional Neural Networks (CNNs), have achieved remarkable success in computer vision tasks like object detection. However, they are computationally intensive and consume significant energy. Spiking Neural Networks (SNNs) offer a promising alternative due to their event-driven nature, leading to lower latency and energy consumption. A common approach to developing SNNs is converting pre-trained CNNs. Yet, this conversion process often suffers from long temporal durations and inference latencies, which degrade accuracy. Furthermore, applying SNNs to complex tasks like object detection, which requires efficient feature extraction and spatiotemporal integration, remains an underexplored area. This paper aims to bridge this gap by proposing a novel SNN architecture that addresses these conversion challenges and enhances object detection performance.
Methods
The core of this work lies in the development and integration of a Bistable Integrate-and-Fire (BIF) neuron model. Unlike traditional Integrate-and-Fire (IF) neurons that reset their membrane potential upon firing, potentially losing information, BIF neurons incorporate a bistability mechanism. This allows them to alternate between spiking and non-spiking states, enabling more reliable and stable spike train encoding.
The authors adapt the widely used ResNet architecture by converting its layers into SNN equivalents using BIF neurons. This conversion is enhanced by:
- Phase Coding: A method to encode activation values into spike trains with higher information density per spike, improving energy efficiency.
- Synchronous Neurons: Employed in residual blocks to ensure that information from shortcut and convolutional paths reaches the output neuron simultaneously, mitigating phase discrepancies and SNN instability.
- SSD Detection Head Fusion: The ResNet backbone is integrated with a Single-Shot MultiBox Detector (SSD) detection head. This allows the network to leverage feature maps generated at various scales for predicting bounding boxes and class labels.
To address class imbalance in object detection, the authors employ focal loss during training, a departure from the traditional hard negative mining used in SSD.
Datasets
The proposed BN-SNN model was evaluated on two datasets:
- MS-COCO (Microsoft Common Objects in Context): A large-scale dataset for object detection, segmentation, and captioning, featuring 81 object categories. The authors used the training data with augmentation and evaluated on the validation set for 80 objects.
- Automotive GEN1 Detection Dataset: An event-based dataset specifically for automotive scenes, containing extensive recordings from an Automotive GEN1 sensor. It includes bounding boxes for cars and pedestrians.
Key Insights and Ideas
- Bistability is Key for SNN Object Detection: The bistable nature of BIF neurons is crucial for balancing temporal encoding and minimizing spike redundancy, leading to improved accuracy and efficiency in object detection.
- Dual BIF Neurons for Robustness: Employing a pair of BIF neurons in a single computational unit ensures continuous information flow and enhances temporal stability, crucial for deep architectures.
- Synchronous Neurons Stabilize ResNet Conversion: Integrating synchronous neurons in the ResNet's residual blocks is vital for aligning information flow between different paths, preventing instability in the converted SNN.
- SSD Integration Enhances Detection: Fusing the BIF-neuron-based ResNet backbone with an SSD detection head effectively leverages multi-scale features for robust object detection.
- Phase Coding Offers Efficiency: This encoding scheme improves information density per spike, contributing to the overall energy efficiency of the SNN.
Key Results
The BN-SNN model demonstrated significant improvements across key metrics:
- MS-COCO Dataset:
- Achieved an [email protected] of 47.6%, outperforming other SNN models.
- Inference time of 36 ms, indicating efficient temporal processing.
- Automotive GEN1 Dataset:
- Achieved a superior [email protected] of 59.1%, the highest among the evaluated models.
- Inference time of 61 ms per frame.
- Lowest firing rate of 21%, suggesting high energy efficiency.
The paper also highlights that a single BIF neuron configuration provided a better balance of accuracy and latency compared to double BIF neuron configurations. Furthermore, BIF neurons outperformed IF and LIF neurons in terms of [email protected] and inference time for object detection.
Limitations and Future Work
- Limitations:
- The paper notes that while the proposed architecture achieves high accuracy and efficiency, further research is needed to deploy these SNNs on low-power hardware for practical embedded applications.
- The study acknowledges that increasing neuron density can introduce phase synchronization challenges, although their proposed architecture mitigates this.
- Future Work:
- Explore the performance of BIF neurons under adversarial conditions, occlusions, and perturbations to validate their generalization capabilities.
- Extend the application of BIF neurons to other areas like semantic segmentation, generative models, and diverse network configurations.
- Investigate their compatibility with deep architectures like ResNet for scalability.
Summary updated