Distributed processing for modelling real-time multimodal perception in a virtual robot

Sylvain Chevallier, Hélène Paugam‐Moisy, François Lemaître

TL;DR

This paper presents a distributed system for a virtual robot that mimics cognitive processes for real-time multimodal perception. By implementing a modular neural network architecture and leveraging parallel processing, the robot can effectively navigate a dynamic prey-predator environment, demonstrating improved survival and exploration capabilities through attentional mechanisms and crossmodal integration.

What Problem Does This Paper Address?

The paper tackles the challenge of creating robots that can perceive and react to their environment in real-time, much like biological organisms. While traditional parallel computing excels at speeding up computations, it doesn't inherently capture the concurrent and distributed nature of cognitive processes in the brain. The authors aim to bridge this gap by designing a robot's "brain" using a modular neural network that simulates multimodal perception (integrating vision and audition) and attentional mechanisms, all within a dynamic virtual environment. This is crucial for developing more sophisticated and adaptable robots capable of complex tasks in unpredictable settings.

Methods

The core of the system is a modular neural network designed to emulate a multimodal associative memory. This network is built from several interconnected neural modules, each responsible for a specific cognitive function:

  • Low-level perception: Separate modules for visual and auditory recognition, using prototype-based incremental classifiers.
  • Central data fusion: A Bidirectional Associative Memory (BAM) module that integrates information from different sensory inputs.
  • High-level identification: An output network that identifies objects based on the fused sensory data.

A key innovation is the parallel implementation of these modules. Instead of sequential processing, all neural network modules run concurrently, allowing for real-time processing of sensory information. This is achieved using a network of PC computers communicating via the Message Passing Interface (MPI).

Furthermore, the paper introduces attentional mechanisms to prioritize sensory information. This involves a "visual buffer" that selects the most salient animal based on pre-attentive cues like predator/prey status and proximity, mimicking how humans focus their attention.

Finally, the system explores crossmodal interactions by incorporating a "spiking-BAM" module. This module uses spiking neurons, which integrate information over time, enabling more dynamic and real-time processing of variable information flows, particularly for integrating auditory and visual cues to detect threats.

Datasets

Not discussed in the paper. The environment is a custom-built virtual zoo with simulated animals.

Key Insights and Ideas

  • Cognitive Architecture for Robotics: The paper demonstrates that a modular neural network architecture, inspired by cognitive processes, can be effectively mapped onto a distributed parallel system for robotic perception.
  • Real-time Attentional Selection: Implementing attentional mechanisms significantly improves a robot's ability to focus on relevant stimuli, leading to better survival and exploration in dynamic environments. This is achieved by prioritizing salient information based on learned cues.
  • Crossmodal Integration for Threat Detection: The integration of auditory and visual information, especially through spiking neural networks, allows the robot to react to threats more effectively, even if the threat is not directly in its primary visual field. This highlights the importance of multisensory processing for robust decision-making.
  • Distributed Processing for Cognitive Tasks: The parallel implementation of cognitive modules is crucial for achieving real-time performance, enabling the robot to react dynamically to its environment rather than processing information in discrete, slow steps.

Key Results

  • Improved Survival: Robots equipped with attentional mechanisms lived significantly longer (Student test, p < 0.05) in the prey-predator environment compared to those without.
  • Enhanced Exploration: Attentive robots made more moves, more forward moves, and visited more distinct places (Student test, p < 0.05), indicating a more efficient exploration strategy.
  • Real-time Reactivity: The distributed system and spiking neural networks enable the robot to process information and react to dynamic environmental changes, such as detecting a predator, in real-time. For instance, the system can modify its internal state based on new sensory input before a stable convergence, allowing for immediate threat avoidance.

Limitations and Future Work

The authors acknowledge that the system, while promising, is not yet directly implementable on a real-world robot. Future work could involve integrating their model with advancements in computer vision, particularly in selecting animals from real-world visual scenes. They also suggest that further research into more sophisticated spiking neural network models and their integration into the distributed architecture could lead to even more performant and realistic cognitive robots.

Summary updated

Abstract

International audience
BibTeX
@inproceedings{chevallier2005distributed,
  author = {Sylvain Chevallier and Hélène Paugam‐Moisy and François Lemaître},
  title = {{Distributed processing for modelling real-time multimodal perception in a virtual robot}},
  booktitle = {HAL},
  year = {2005},
}
Figure