A toolbox for neuromorphic perception in robotics

Julien Dupeyroux, Stein Stroobants, Guido C. H. E. de Croon

TL;DR

This paper introduces a comprehensive toolbox for converting traditional sensor data into the "spike" format required by neuromorphic AI systems, a crucial step for integrating these energy-efficient and robust AI approaches into robotics. They review and benchmark various spike coding algorithms and provide a ROS package to enable real-time neuromorphic perception with off-the-shelf sensors.

What Problem Does This Paper Address?

Neuromorphic AI, particularly Spiking Neural Networks (SNNs), promises significant advantages for robotics, including high flexibility, energy efficiency, and robustness. However, a major hurdle is the incompatibility between most existing robotic sensors (like IMUs, sonars, radars) and the spike-based nature of neuromorphic systems. Traditional AI, especially deep learning, is computationally intensive and often not suitable for real-time robotic applications. To bridge this gap, sensor data needs to be converted into spike streams (events with timestamps and polarity). This paper addresses the need for standardized, well-understood, and easily implementable methods for this conversion, specifically for robotics applications.

Methods

The authors provide a review and benchmark of various spike coding algorithms, categorizing them into three main types:

  • Population Coding: Information is encoded by the activity of a group of neurons, each responding to a specific range of the input signal. Two models are presented:
  • Position Coding: A simple linear division of the input distribution among neurons.
  • Gaussian Receptive Fields (GRF): Neurons have Gaussian-shaped response distributions, allowing for more nuanced encoding.
  • Rate Coding: Information is encoded in the frequency of spikes. Several algorithms are discussed:
  • Hough Spike Algorithm (HSA): Uses a reverse convolution with a filter and a threshold.
  • Threshold Hough Spike Algorithm (T-HSA): Similar to HSA but with a dynamic threshold.
  • Ben's Spike Algorithm (BSA): Employs a reverse convolution and compares two error metrics.
  • Temporal Coding: Information is encoded in the precise timing of spikes. Algorithms include:
  • Temporal-Based Representation (TBR): Spikes are emitted when signal variations exceed a threshold (similar to event-based cameras).
  • Step Forward Algorithm (SF): Uses a baseline signal and a threshold to detect variations.
  • Moving Window Algorithm (MW): Similar to SF but uses a moving average as the baseline.

The paper also introduces a ROS (Robot Operating System) toolbox that implements these encoding and decoding algorithms in Python. This toolbox includes custom ROS messages for spike data and a node that can be launched to process sensor inputs in real-time, publishing spike streams and recording data.

Datasets

The paper does not explicitly mention training datasets. For evaluation, they use 1D signals generated as sums of noisy sine waves. They also provide an example of applying the SF algorithm to a video stream from the open-source Obstacle and Avoidance (ODA) Dataset.

Key Insights and Ideas

  • Bridging the Sensor Gap: The core idea is that converting traditional sensor data into spikes is essential for leveraging the benefits of neuromorphic computing in robotics.
  • Algorithm Trade-offs: Different coding schemes excel in different aspects. Temporal coding offers high precision but can be sensitive to noise, while rate coding is more robust but can be less efficient in terms of spike count. Population coding offers a balance.
  • ROS Integration is Key: Making these algorithms readily available within the ROS framework significantly lowers the barrier to entry for roboticists wanting to experiment with neuromorphic AI.
  • Performance vs. Efficiency: There's a clear trade-off between the quality of signal reconstruction (low RMSE) and spiking efficiency (fewer spikes). The SF algorithm stands out for its high spiking efficiency and consistent reconstruction quality.
  • Temporal Coding's Drift: Temporal coding schemes like TBR and MW can accumulate errors over time, leading to a "drift" in the reconstructed signal, especially over longer durations.

Key Results

  • Spiking Efficiency: The Step Forward (SF) algorithm consistently achieved the highest spiking efficiency (around 73%), largely unaffected by signal duration. Rate coding algorithms (HSA, T-HSA) also showed good efficiency (>50%), while TBR and MW were less efficient.
  • Reconstruction Quality (RMSE): The SF algorithm demonstrated the best reconstruction quality with a very low average RMSE (0.26 ± 0.01) across various signal durations. Rate coding algorithms also showed good RMSE, improving with signal duration. Temporal coding algorithms (TBR, MW) showed increasing RMSE over time.
  • ROS Toolbox: The developed ROS toolbox allows for real-time encoding and decoding of sensor data, demonstrated with a simulated 1D signal and a video input example.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the performance of rate coding schemes is dependent on the chosen filter, which is case-specific.
  • Temporal coding schemes can suffer from signal drift due to cumulative errors.
  • The GRF population coding scheme, while accurate, can be computationally expensive and has null spiking efficiency (spikes at every timestamp).
  • Future Work:
    • The authors plan to integrate more coding schemes into the toolbox, including non-linear GRF, Rank Order Coding (ROC), and Latency-Phase temporal Coding (LPC).
  • They suggest using evolutionary algorithms (like DEAP, PyBrain) for automatically determining the optimal parameters for these coding schemes in real-world robotic applications.

Summary updated

Abstract

The third generation of artificial intelligence (AI) introduced by neuromorphic computing is revolutionizing the way robots and autonomous systems can sense the world, process the information, and interact with their environment. Research towards fulfilling the promises of high flexibility, energy efficiency, and robustness of neuromorphic systems is widely supported by software tools for simulating spiking neural networks, and hardware integration (neuromorphic processors). Yet, while efforts have been made on neuromorphic vision (event-based cameras), it is worth noting that most of the sensors available for robotics remain inherently incompatible with neuromorphic computing, where information is encoded into spikes. To facilitate the use of traditional sensors, we need to convert the output signals into streams of spikes, i.e., a series of events (+1,-1) along with their corresponding timestamps. In this paper, we propose a review of the coding algorithms from a robotics perspective and further supported by a benchmark to assess their performance. We also introduce a ROS (Robot Operating System) toolbox to encode and decode input signals coming from any type of sensor available on a robot. This initiative is meant to stimulate and facilitate robotic integration of neuromorphic AI, with the opportunity to adapt traditional off-the-shelf sensors to spiking neural nets within one of the most powerful robotic tools, ROS.
BibTeX
@inproceedings{dupeyroux2022toolbox,
  author = {Julien Dupeyroux and Stein Stroobants and Guido C. H. E. de Croon},
  title = {{A toolbox for neuromorphic perception in robotics}},
  booktitle = {EBCCSP},
  year = {2022},
  doi = {10.1109/ebccsp56922.2022.9845664},
}
Figure