CarSNN: An Efficient Spiking Neural Network for Event-Based Autonomous Cars on the Loihi Neuromorphic Research Processor

Alberto Viale, Alberto Marchisio, Maurizio Martina, Guido Masera, Muhammad Shafique

Abstract

CarSNN is a pioneering implementation of an event-based car classifier on Intel’s Loihi neuromorphic chip. By leveraging bio-inspired spiking neural networks (SNNs) and event-based cameras, the system achieves sub-millisecond latency (0.72 ms) and ultra-low power consumption (314.5 mW). This work demonstrates that autonomous driving tasks can be performed with high energy efficiency and responsiveness, addressing the power and latency bottlenecks of traditional frame-based deep learning.

Introduction

Autonomous Driving (AD) presents a significant computational challenge for mobile systems. Real-time navigation requires near-zero latency, yet traditional Deep Learning (DL) typically relies on power-hungry GPUs and frame-based cameras. These cameras produce redundant data by capturing static scenes at fixed intervals, leading to unnecessary processing and energy drain—a critical issue for battery-constrained electric vehicles and small robots.

CarSNN addresses these challenges by adopting neuromorphic computing. The system focuses on the binary classification of "cars vs. background," aiming to provide a reactive and lean solution suitable for high-speed driving on a limited power budget.

Methods

The researchers developed CarSNN, a Spiking Convolutional Neural Network (SCNN), based on three core pillars:

  1. Event-Based Vision: Instead of standard frames, the system uses a Dynamic Vision Sensor (DVS). Similar to the human retina, these sensors only transmit data when a pixel detects a change in brightness, naturally filtering out static backgrounds and reducing data volume.
  2. Attention Window Strategy: Analysis of the N-CARS dataset revealed that critical information is often concentrated in specific regions. The authors implemented an "attention window" to crop inputs (e.g., to 50x50 or 100x100 pixels), reducing the neuron count without significantly impacting accuracy.
Heatmaps illustrating the attention window strategy used to focus on high-information regions in the N-CARS dataset.
  1. Spatio-Temporal Backpropagation (STBP): Because spikes are non-differentiable, the authors used STBP to train the network offline. The resulting weights and thresholds were then mapped onto the Intel Loihi research chip, converting 64-bit floating-point values into 8-bit fixed-point integers.

Datasets

The study utilized the N-CARS dataset, a benchmark for event-based object recognition containing ~24,000 samples (100ms each) recorded from a car windshield using an ATIS (Asynchronous Time-based Image Sensor). This dataset provides a realistic representation of driving conditions, including varied lighting and weather.

Key Results

The implementation on the Intel Loihi chip demonstrated significant advantages in both speed and efficiency:

  • Speed: The system achieves a maximum latency of 0.72 ms per sample, significantly faster than the ~16ms latency of standard 60fps cameras.
  • Efficiency: The Loihi neuro-cores consume only 314.5 mW, with an energy cost of approximately 320 µJ per inference.
  • Accuracy: The model achieved 86% accuracy offline and 83% on the Loihi hardware. The slight drop is attributed to fixed-point arithmetic constraints, yet it remains competitive with other state-of-the-art architectures.
Comparison of accuracy and latency across various neuromorphic and traditional architectures.

Discussion and Future Work

CarSNN proves the viability of on-chip car classification using neuromorphic hardware. However, several limitations remain. The current attention window is static, whereas real-world scenarios require dynamic or "foveated" attention as objects move. Additionally, the task is currently limited to binary classification; future work must expand to multi-class detection (pedestrians, signs) and localization. Finally, leveraging Loihi’s on-chip learning capabilities could allow vehicles to adapt to new environments in real-time.

Summary updated

Abstract

Autonomous Driving (AD) related features provide new forms of mobility that are also beneficial for other kind of intelligent and autonomous systems like robots, smart transportation, and smart industries. For these applications, the decisions need to be made fast and in real-time. Moreover, in the quest for electric mobility, this task must follow low power policy, without affecting much the autonomy of the mean of transport or the robot. These two challenges can be tackled using the emerging Spiking Neural Networks (SNNs). When deployed on a specialized neuromorphic hardware, SNNs can achieve high performance with low latency and low power consumption. In this paper, we use an SNN connected to an event-based camera for facing one of the key problems for AD, i.e., the classification between cars and other objects. To consume less power than traditional frame-based cameras, we use a Dynamic Vision Sensor (DVS). The experiments are made following an offline supervised learning rule, followed by mapping the learnt SNN model on the Intel Loihi Neuromorphic Research Chip. Our best experiment achieves an accuracy on offline implementation of 86%, that drops to 83% when it is ported onto the Loihi Chip. The Neuromorphic Hardware implementation has maximum 0.72 ms of latency for every sample, and consumes only 310 mW. To the best of our knowledge, this work is the first implementation of an event-based car classifier on a Neuromorphic Chip.
BibTeX
@inproceedings{viale2021carsnn,
  author = {Alberto Viale and Alberto Marchisio and Maurizio Martina and Guido Masera and Muhammad Shafique},
  title = {{CarSNN: An Efficient Spiking Neural Network for Event-Based Autonomous Cars on the Loihi Neuromorphic Research Processor}},
  booktitle = {IJCNN},
  year = {2021},
  doi = {10.1109/ijcnn52387.2021.9533738},
}
Figure