Simultaneous neuromorphic selection of multiple salient objects for event vision

Amélie Gruel, Jean Martinet, Michele MagnoIJCNN 2023 · DOI

TL;DR

This paper introduces a novel neuromorphic model that can simultaneously identify and select multiple "objects of interest" (OoIs) from event-based camera data. This approach significantly reduces the amount of data processed by embedded systems while retaining high classification accuracy, paving the way for more efficient and biologically plausible visual processing in real-time applications.

What Problem Does This Paper Address?

Event cameras, which capture changes in pixel intensity asynchronously, are highly promising for embedded computer vision due to their low latency and power consumption. However, processing the dense stream of events generated by these cameras, especially in dynamic scenes, can overwhelm the limited resources of embedded systems. Existing methods often struggle to balance data reduction with information preservation. This paper tackles the challenge of efficiently processing event data by focusing on identifying and extracting only the most salient objects within a scene, a process inspired by human visual attention. The goal is to reduce the computational burden without sacrificing critical visual information needed for downstream tasks like classification.

Methods

The core of this work is an extension of a previously developed neuromorphic saliency detection model. The key innovations include:

  • Simultaneous Multi-OoI Selection: The model can now identify and select multiple objects of interest (OoIs) concurrently, rather than just one. This is achieved by extending the output layers of the architecture.
  • Intrinsic SNN Dynamics: The model leverages the inherent dynamics of Spiking Neural Networks (SNNs) and adaptive synaptic/neuronal rules, eliminating the need for explicit training phases or external supervisors. This allows for real-time processing without the overhead of traditional machine learning training.
  • Dynamic Threshold Adaptation: A novel dynamic threshold adaptation rule is introduced. This rule dynamically adjusts neuronal thresholds based on both the saliency detector's output and the activity of lateral output layers. This mechanism is crucial for enabling the simultaneous selection of multiple OoIs while managing their interactions.
  • Bio-inspired Saliency: The saliency detection itself is based on identifying regions with high spatiotemporal density of events, mimicking how biological systems prioritize visually salient areas.

The underlying SNN model used is the "Leaky Integrate-And-Fire" (LIF) neuron, chosen for its simplicity and efficiency.

Datasets

Since a suitable dataset for evaluating multi-OoI selection in event-based vision was not readily available, the authors created custom datasets:

  • "Control" Dataset: Created by combining multiple samples from the DVS 128 Gesture dataset side-by-side, with controlled spatial offsets. This dataset allows for the detection of a known number of OoIs.
  • "Random Symmetric" Dataset: Similar to the "control" dataset, but with permutations of sample order to create a more randomized and symmetric dataset. This helps in evaluating the model's robustness to variations in event data.

Both custom datasets were derived from the DVS 128 Gesture dataset, a standard benchmark for event-based gesture recognition. For experimental purposes, the input data was spatially downscaled (by a factor of 4) and only the first 100ms of each sample was used to manage computational load.

Key Insights and Ideas

  • Bio-plausibility of Event Density: The research strongly suggests that prioritizing event density for saliency detection is a bio-plausible approach, mirroring human visual attention mechanisms.
  • Efficiency through Selection: By focusing on selecting only salient objects, the model significantly reduces data volume while preserving crucial information for downstream tasks, demonstrating a powerful trade-off for embedded systems.
  • The Power of Intrinsic Dynamics: The reliance on SNNs' intrinsic dynamics for adaptation and selection highlights the potential of neuromorphic computing for low-power, real-time processing without extensive training.
  • Parameter Tuning is Crucial: The performance of multi-OoI selection is sensitive to the tuning of synaptic and neuronal parameters, emphasizing the need for careful hyperparameter optimization in SNNs.

Key Results

  • Latency: The proposed multi-OoI selection model maintains a low latency, with a maximum of 14ms without shift and decreasing to 5ms with increasing temporal shifts between OoIs. This is comparable to their previous single-OoI model.
  • Classification Accuracy: The filtered OoIs retain a significant portion of the original data's classification performance, reaching up to 73% of the original accuracy.
  • Data Reduction: The model achieves substantial data reduction, with activated surface reduced by up to 80% and a significant reduction in spatiotemporal densities.
  • Efficiency: The model uses a reduced number of neurons and synapses compared to existing state-of-the-art methods, making it suitable for embedded deployment.
  • No Training Required: The model operates without a training phase, relying solely on intrinsic SNN dynamics.

Limitations and Future Work

  • Dataset Limitations: The authors acknowledge the lack of existing datasets specifically designed for multi-OoI selection in event-based vision, necessitating the creation of custom datasets.
  • Parameter Sensitivity: While the model demonstrates good performance, the sensitivity to parameter tuning (e.g., WWTA, Δθ, ts) suggests that further research into automated parameter optimization or more robust adaptation rules could be beneficial.
  • Future Directions: The authors plan to validate their architecture on larger event-based datasets (e.g., One Megapixel Detection Dataset), extend it to handle an even greater number of OoIs, and implement it on neuromorphic hardware like SpiNNaker or Kraken for real-world embedded applications. They also see potential for its use in multi-object tracking and scene segmentation.

Summary updated

BibTeX
@inproceedings{gruel2023simultaneous,
  author = {Amélie Gruel and Jean Martinet and Michele Magno},
  title = {{Simultaneous neuromorphic selection of multiple salient objects for event vision}},
  booktitle = {IJCNN},
  year = {2023},
  doi = {10.1109/ijcnn54540.2023.10191543},
}
Figure