Performance Comparison of DVS Data Spatial Downscaling Methods Using Spiking Neural Networks

Amélie Gruel, Jean Martinet, Bernabé Linares-Barranco, Teresa Serrano-GotarredonaWACV 2023

TL;DR

This paper introduces two novel Spiking Neural Network (SNN) architectures for spatially downscaling event data from Dynamic Vision Sensors (DVS). These SNN-based methods offer a more efficient and information-preserving way to reduce the high data rates of DVS cameras, outperforming existing methods in both classification tasks and real-time processing on neuromorphic hardware.

What Problem Does This Paper Address?

Event cameras, like DVS, offer significant advantages over traditional cameras, including high temporal resolution, low power consumption, and high dynamic range. However, they generate a massive amount of sparse, asynchronous event data. This high data rate poses a significant challenge for embedded systems with limited resources (energy, memory, computation, bandwidth). Simply dropping events to reduce data can lead to loss of crucial information. Therefore, there's a critical need for intelligent methods to reduce this event data stream while preserving its essential information for downstream tasks. This paper addresses this by proposing and evaluating SNN-based spatial downscaling techniques.

Methods

The authors propose two novel SNN architectures for spatial downscaling, building upon the concept of SNN pooling. These methods leverage the intrinsic dynamics of Leaky Integrate-and-Fire (LIF) neurons, which mimic the behavior of DVS pixels.

  • Separate SNN: This architecture handles positive and negative events independently. Input events are mapped to corresponding LIF neurons in a 2D input layer, which then activate neurons in a smaller output layer. This method treats positive and negative events as distinct, without considering their interaction.
  • Mutual SNN: This approach addresses a limitation of the "separate" method by incorporating mutual inhibition between positive and negative event processing. When a positive event activates a neuron, it also inhibits the neuron processing negative events in the same spatial region, and vice-versa. This better reflects the physics of event cameras and leads to a more stable output, akin to a downscaled log-luminance signal.

Both methods are implemented using the PyNN simulator and adapted for neuromorphic hardware like SpiNNaker.

Datasets

The paper uses two established event-based datasets for evaluation:

  • DVS128 Gesture dataset: This dataset features hand gestures recorded with a DVS128 camera, commonly used for event data classification.
  • Neuromorphic-MNIST (N-MNIST): This is an event-based translation of the MNIST dataset, known for its high event density.

Key Insights and Ideas

  • SNNs as a natural fit for event data reduction: The asynchronous and sparse nature of SNNs makes them inherently well-suited for processing event-based data, allowing for efficient downscaling without explicit frame reconstruction.
  • Mutual inhibition is crucial for realistic downscaling: The "Mutual SNN" architecture, by incorporating inhibition between positive and negative event pathways, more accurately mimics the behavior of a downscaled DVS sensor and achieves a better trade-off between event reduction and information preservation.
  • Hyperparameter tuning is key: Similar to DVS camera settings, tuning hyperparameters like synaptic weight in SNNs is critical for achieving optimal downscaling performance and balancing information retention with data reduction.
  • Neuromorphic hardware is essential for real-time processing: While SNN simulations on CPUs are slow, the paper highlights the potential of neuromorphic hardware like SpiNNaker for achieving near real-time event data downscaling.

Key Results

  • Improved classification performance: The proposed SNN-based downscaling methods, particularly the "Mutual SNN," generally lead to better classification accuracy compared to other methods (spatial funnelling, event count, log-luminance reconstruction) on both DVS128 Gesture and N-MNIST datasets, especially when considering the number of retained events.
  • Efficient data reduction: The "Mutual SNN" method effectively reduces the number of events and temporal density, offering a good compromise between data reduction and information content.
  • Near real-time processing on neuromorphic hardware: When implemented on SpiNNaker, the SNN pooling methods demonstrate significantly faster processing times compared to other methods, though still not strictly real-time on the tested hardware.
  • "Spatial funnelling" is suboptimal: This method consistently retains a high number of events and shows poor performance trade-offs, even with increased dividing factors.

Limitations and Future Work

  • Real-time processing gap: Despite improvements, the SNN methods are still "two orders of magnitude slower than real time" on the tested CPU hardware. Achieving true real-time performance likely requires more advanced neuromorphic hardware or further algorithmic optimization.
  • Hyperparameter sensitivity: The performance of SNN downscaling is sensitive to hyperparameter tuning, which can be dataset-dependent.
  • Future work: The authors suggest exploring event-based foveation mechanisms and temporal downscaling techniques as future research directions. Temporal downscaling could involve changing timestamp granularity or selecting a subset of events.

Summary updated

BibTeX
@inproceedings{gruel2023performance,
  author = {Amélie Gruel and Jean Martinet and Bernabé Linares-Barranco and Teresa Serrano-Gotarredona},
  title = {{Performance Comparison of DVS Data Spatial Downscaling Methods Using Spiking Neural Networks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure