Abstract
CarSNN is a pioneering implementation of an event-based car classifier on Intel’s Loihi neuromorphic chip. By leveraging bio-inspired spiking neural networks (SNNs) and event-based cameras, the system achieves sub-millisecond latency (0.72 ms) and ultra-low power consumption (314.5 mW). This work demonstrates that autonomous driving tasks can be performed with high energy efficiency and responsiveness, addressing the power and latency bottlenecks of traditional frame-based deep learning.
Introduction
Autonomous Driving (AD) presents a significant computational challenge for mobile systems. Real-time navigation requires near-zero latency, yet traditional Deep Learning (DL) typically relies on power-hungry GPUs and frame-based cameras. These cameras produce redundant data by capturing static scenes at fixed intervals, leading to unnecessary processing and energy drain—a critical issue for battery-constrained electric vehicles and small robots.
CarSNN addresses these challenges by adopting neuromorphic computing. The system focuses on the binary classification of "cars vs. background," aiming to provide a reactive and lean solution suitable for high-speed driving on a limited power budget.
Methods
The researchers developed CarSNN, a Spiking Convolutional Neural Network (SCNN), based on three core pillars:
- Event-Based Vision: Instead of standard frames, the system uses a Dynamic Vision Sensor (DVS). Similar to the human retina, these sensors only transmit data when a pixel detects a change in brightness, naturally filtering out static backgrounds and reducing data volume.
- Attention Window Strategy: Analysis of the N-CARS dataset revealed that critical information is often concentrated in specific regions. The authors implemented an "attention window" to crop inputs (e.g., to 50x50 or 100x100 pixels), reducing the neuron count without significantly impacting accuracy.
- Spatio-Temporal Backpropagation (STBP): Because spikes are non-differentiable, the authors used STBP to train the network offline. The resulting weights and thresholds were then mapped onto the Intel Loihi research chip, converting 64-bit floating-point values into 8-bit fixed-point integers.
Datasets
The study utilized the N-CARS dataset, a benchmark for event-based object recognition containing ~24,000 samples (100ms each) recorded from a car windshield using an ATIS (Asynchronous Time-based Image Sensor). This dataset provides a realistic representation of driving conditions, including varied lighting and weather.
Key Results
The implementation on the Intel Loihi chip demonstrated significant advantages in both speed and efficiency:
- Speed: The system achieves a maximum latency of 0.72 ms per sample, significantly faster than the ~16ms latency of standard 60fps cameras.
- Efficiency: The Loihi neuro-cores consume only 314.5 mW, with an energy cost of approximately 320 µJ per inference.
- Accuracy: The model achieved 86% accuracy offline and 83% on the Loihi hardware. The slight drop is attributed to fixed-point arithmetic constraints, yet it remains competitive with other state-of-the-art architectures.
Discussion and Future Work
CarSNN proves the viability of on-chip car classification using neuromorphic hardware. However, several limitations remain. The current attention window is static, whereas real-world scenarios require dynamic or "foveated" attention as objects move. Additionally, the task is currently limited to binary classification; future work must expand to multi-class detection (pedestrians, signs) and localization. Finally, leveraging Loihi’s on-chip learning capabilities could allow vehicles to adapt to new environments in real-time.
Summary updated