Multimodal Signal Processing and Learning Aspects of Human-Robot Interaction for an Assistive Bathing Robot

Athanasia Zlatintsi, Isidoros Rodomagoulakis, Petros Koutras, A. C. Dometios, Vassilis Pitsikalis, Costas S. Tzafestas, Petros Maragos

Abstract

We explore new aspects of assistive living on smart human-robot interaction (HRI) that involve automatic recognition and online validation of speech and gestures in a natural interface, providing social features for HRI. We introduce a whole framework and resources of a real-life scenario for elderly subjects supported by an assistive bathing robot, addressing health and hygiene care issues. We contribute a new dataset and a suite of tools used for data acquisition and a state-of-the-art pipeline for multimodal learning within the framework of the I-Support bathing robot, with emphasis on audio and RGB-D visual streams. We consider privacy issues by evaluating the depth visual stream along with the RGB, using Kinect sensors. The audio-gestural recognition task on this new dataset yields up to 84.5%, while the online validation of the I-Support system on elderly users accomplishes up to 84% when the two modalities are fused together. The results are promising enough to support further research in the area of multimodal recognition for assistive social HRI, considering the difficulties of the specific task.

Introduction

As the global population ages, the demand for assistive technologies in healthcare and hygiene is increasing rapidly. Bathing is a fundamental Activity of Daily Living (ADL), yet it remains a highly private and physically challenging task for many elderly individuals. While robotic systems offer a potential solution, the primary bottleneck remains the interface. Traditional buttons or touchscreens are often impractical in wet, soapy environments. This research addresses the challenge of creating a natural Human-Robot Interaction (HRI) system for the I-Support bathing robot, capable of understanding simultaneous speech and gestures while maintaining user privacy.

Methodology

The proposed framework utilizes a dual-stream pipeline to process audio and visual signals, enabling robust command recognition in noisy environments.

Visual Stream and Privacy-Preserving Vision

The system employs Kinect sensors to capture RGB and Depth data. To extract motion features, the researchers use Dense Trajectories, which track feature points through time using optical flow. These trajectories are described using Histograms of Oriented Gradients (HOG), Histograms of Optical Flow (HOF), and Motion Boundary Histograms (MBH). Features are encoded using VLAD (Vector of Locally Aggregated Descriptors), which was found to outperform standard Bag-of-Visual-Words (BoVW) by capturing richer information about the distribution of visual descriptors.

Sensor placement and viewing angles for the I-Support system.

A critical technical contribution is the use of log-depth transformations. By applying a non-linear log transform to depth data, the system enhances edges related to hand movements while obscuring sensitive personal details. This approach addresses the "privacy vs. utility" trade-off, making the system more acceptable for use in a bathroom setting compared to raw RGB video.

Audio Stream and Multimodal Fusion

The audio stream processes far-field signals from eight distributed microphones. The system uses a task-dependent grammar consisting of 23 commands in German and Italian. To handle individual variations in speech, Maximum Likelihood Linear Regression (MLLR) is used to adapt acoustic models to specific users.

The two streams are combined via late fusion at the decision level. If the top speech hypothesis matches one of the top two gesture hypotheses, the system confirms the command. This redundancy acts as a safety net; if speech is muffled by running water or a gesture is partially occluded by the robot arm, the other modality can often successfully resolve the command.

Experimental Setup

The study utilized a new multimodal development dataset featuring 23 users performing 25–27 gestures and 8 users providing spoken commands. Real-world validation was conducted with 54 elderly participants (mean age ~67–81) at two clinical sites: the Fondazione Santa Lucia Hospital in Italy and the Bethanien Hospital in Germany.

The experimental setup featuring the motorized chair, soft robotic arm, and distributed Kinect sensors.

Results and Discussion

The experimental results demonstrate the efficacy of the multimodal approach:

  • Recognition Accuracy: Combined audio-gestural recognition reached 84.5% on the development dataset and up to 84% during online validation in real hospital environments.
  • Modality Performance: Elderly participants achieved over 90% accuracy with spoken commands, while gesture performance ranged between 70% and 80%.
  • User Adaptation: Performance improved over the course of the trials, suggesting the interface is intuitive enough for seniors to master quickly. However, the researchers noted that performing simultaneous speech and gestures increased cognitive load, slightly degrading performance compared to single-modality tasks.

Conclusion

The I-Support system provides a viable framework for privacy-conscious, natural HRI in assistive bathing. While occlusions from the robot arm or chair remain a challenge for gesture recognition, the multimodal fusion significantly improves robustness. Future work will explore deep learning architectures to reduce response times and investigate "co-speech gesturing" to create even more fluid, human-like interactions.

Summary updated

Abstract

We explore new aspects of assistive living on smart human-robot interaction (HRI) that involve automatic recognition and online validation of speech and gestures in a natural interface, providing social features for HRI. We introduce a whole framework and resources of a real-life scenario for elderly subjects supported by an assistive bathing robot, addressing health and hygiene care issues. We contribute a new dataset and a suite of tools used for data acquisition and a state-of-the-art pipeline for multimodal learning within the framework of the I-Support bathing robot, with emphasis on audio and RGB-D visual streams. We consider privacy issues by evaluating the depth visual stream along with the RGB, using Kinect sensors. The audio-gestural recognition task on this new dataset yields up to 84.5%, while the online validation of the I-Support system on elderly users accomplishes up to 84% when the two modalities are fused together. The results are promising enough to support further research in the area of multimodal recognition for assistive social HRI, considering the difficulties of the specific task. Upon acceptance of the paper part of the data will be publicly available.
BibTeX
@inproceedings{zlatintsi2018multimodal,
  author = {Athanasia Zlatintsi and Isidoros Rodomagoulakis and Petros Koutras and A. C. Dometios and Vassilis Pitsikalis and Costas S. Tzafestas and Petros Maragos},
  title = {{Multimodal Signal Processing and Learning Aspects of Human-Robot Interaction for an Assistive Bathing Robot}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=UwKjLzYrXN},
}
Figure