A Multi-modal Approach for Assistive Humanoid Robots

German Ignacio Parisi, Johannes Bauer, Erik Strahl, Stefan WermterMuSRobS @ IROS 2015 · OpenReview

Abstract As the global population ages, there is an increasing demand for technologies that support elderly independence. This paper presents a multi-modal framework for the Nao humanoid robot designed to detect falls and provide interactive assistance. By integrating active visual tracking, sound source localization, and speech recognition, the system overcomes the limitations of static sensors and narrow-field-of-view cameras. Experimental results show a 96% fall detection accuracy and an expanded monitoring range of 138°, demonstrating a robust approach to non-invasive robotic companionship in domestic settings.

Introduction

Falls are a leading cause of injury and loss of independence among the elderly. While existing solutions like wall-mounted cameras or wearable sensors exist, they are often invasive, expensive, or limited by fixed perspectives. Mobile robots offer a flexible alternative but struggle with "sensor uncertainty," such as losing track of a person when they move out of view. This research addresses these challenges by creating a robust multi-modal controller that fuses vision, sound, and sonar data to maintain continuous user monitoring in complex home environments.

System Architecture and Methods

The researchers extended a standard Aldebaran Nao robot with an ASUS Xtion Pro depth sensor and Soundman OKM II binaural stereo microphones. The system architecture integrates several key modalities:

  • Active Visual Tracking: To compensate for the depth sensor's 58° horizontal field of view, the robot uses its head actuators to pan the sensor, effectively expanding the monitoring range to 138°.
  • Fall Detection: A neural network trained on 3D body motion data distinguishes between normal activities (e.g., walking, sitting) and falls.
  • Sound Source Localization (SSL): When a user is outside the visual field, the system uses Time Difference of Arrival (TDOA) between the microphones to re-orient the robot's head toward the source of speech.
  • Automatic Speech Recognition (ASR): Vocal commands are processed via a cloud-based service and a domain-specific post-processor to ensure reliable interaction.
  • Multi-modal Controller: This central unit manages sensor redundancy. It prioritizes visual data for positioning but switches to SSL if visual contact is lost for more than three seconds.
System architecture and hardware integration

Results and Discussion

The system was evaluated using various benchmarks, including the TIMIT speech corpus and a custom 3D body motion dataset. Key results include:

  • Fall Detection: The system achieved a 96% accuracy rate in detecting falls within a home-like environment.
  • Field of View Expansion: Active tracking successfully increased the horizontal monitoring range from 58° to 138°.
  • Sensor Performance: External microphones significantly outperformed the robot's built-in hardware for localization.
  • Safety Features: To reduce false alarms, the system records five seconds of video preceding a detected fall, which can be transmitted to a human caregiver for verification.
Active tracking strategy and depth-based fall detection sequence

Conclusion

This multi-modal approach demonstrates the potential for humanoid robots to provide reliable, non-invasive assistance for the elderly. By combining active perception with redundant sensory streams, the system maintains a high degree of reliability even in dynamic environments. Future work will focus on moving all processing onboard, integrating SLAM for autonomous navigation between rooms, and conducting long-term usability studies with elderly participants.

Summary updated

BibTeX
@inproceedings{parisi2015multi,
  author = {German Ignacio Parisi and Johannes Bauer and Erik Strahl and Stefan Wermter},
  title = {{A Multi-modal Approach for Assistive Humanoid Robots}},
  booktitle = {MuSRobS @ IROS},
  year = {2015},
  url = {https://openreview.net/forum?id=YSewbCjcMF},
}
Figure