Multimodal Signal Processing and Learning Aspects of Human-Robot Interaction for an Assistive Bathing Robot

Athanasia Zlatintsi, Isidoros Rodomagoulakis, Petros Koutras, A. C. Dometios, Vassilis Pitsikalis, Costas S. Tzafestas, Petros Maragos

TL;DR

Researchers have developed a multimodal interaction framework for I-Support, a robotic system designed to assist the elderly with bathing. By fusing speech recognition with RGB-D gesture tracking, the system achieves over 84% command accuracy in real-world hospital settings, providing a natural and privacy-conscious way for seniors to maintain their independence.

What Problem Does This Paper Address?

As the global population ages, the demand for assistive technologies in healthcare and hygiene is skyrocketing. Bathing is a fundamental "Activity of Daily Living" (ADL), yet it remains a highly private and physically challenging task for many elderly individuals.

While robots can help, the interface is the bottleneck. Traditional buttons or touchscreens are often impractical in a wet, soapy environment. This paper addresses the challenge of creating a natural Human-Robot Interaction (HRI) system that can understand both what a user says and what they do (gestures) simultaneously. Furthermore, it tackles the "privacy vs. utility" trade-off: how do you use cameras to track a person in a bathroom without violating their dignity?

Methods

The authors propose a dual-stream pipeline for processing audio and visual signals:

  • Visual Stream (Gestures): The system uses Kinect sensors to capture RGB and Depth data. To extract features, they employ Dense Trajectories, which sample and track feature points through time using optical flow. These are described using HOG (Histograms of Oriented Gradients), HOF (Histograms of Optical Flow), and MBH (Motion Boundary Histograms).
  • Encoding & Classification: Features are encoded using both Bag-of-Visual-Words (BoVW) and VLAD (Vector of Locally Aggregated Descriptors), then classified using Support Vector Machines (SVMs).
  • Privacy-Preserving Vision: A key technical contribution is the use of log-depth transformations. By applying a non-linear log transform to depth data, the system enhances edges related to hand movements while obscuring sensitive personal details, making it more "privacy-aware" than raw RGB.
  • Audio Stream (Speech): The system processes far-field signals from 8 distributed microphones. It uses a task-dependent grammar (23 commands in German/Italian) and employs Maximum Likelihood Linear Regression (MLLR) to adapt acoustic models to individual users.
  • Late Fusion: The two streams are combined at a decision level. If the top speech hypothesis matches one of the top two gesture hypotheses, the system confirms the command.

Datasets

The study utilizes two primary data sources:

  1. A New Multimodal Development Dataset: Created specifically for this project, featuring 23 users performing 25–27 different gestures and 8 users providing 23 spoken German commands.
  2. Real-World Validation Data: Collected from 54 elderly participants (mean age ~67–81) at two clinical sites: the Fondazione Santa Lucia Hospital in Italy and the Bethanien Hospital in Germany.

Key Insights and Ideas

  • The "Privacy-Utility" Balance: While RGB data consistently yields higher accuracy (~85%), the researchers found that log-depth is a viable alternative. It provides enough detail for motion tracking while offering a layer of visual abstraction that users may find more acceptable in a bathroom setting.
  • Multimodality as a Safety Net: The "always-listening" nature of the system means that if a user’s speech is muffled by running water or their gesture is partially occluded by the robot arm, the other modality can often "rescue" the command.
  • The Learning Curve: Interestingly, user performance improved over the course of the trials. Initial confusion (especially with gesture ID 5) gave way to high success rates (over 90% for "Halt" commands), suggesting that the interface is intuitive enough for seniors to master quickly.
  • Cognitive Load Matters: The researchers noted that performing simultaneous speech and gestures increased the cognitive load on elderly users, slightly degrading their performance compared to single-modality tasks.

Key Results

  • Offline Accuracy: The combined audio-gestural recognition reached 84.5% on the development dataset.
  • Online Validation: In real hospital environments, the Multimodal Command Recognition Rate (MCRR) reached up to 84%.
  • User Performance: Elderly participants were highly successful with spoken commands (over 90% accuracy), while gesture performance was satisfactory, ranging between 70% and 80%.
  • Encoding Comparison: VLAD encoding consistently outperformed BoVW because it captures richer information about the distribution of visual words.

Limitations and Future Work

The authors identify occlusions as a primary hurdle; when the robot arm or the user's chair blocks the camera's view of the hands, gesture recognition fails. There were also a small number of "false alarms" where the system triggered without a deliberate command.

Future directions include: * Integrating Deep Learning architectures to improve robustness and reduce response times. * Exploring co-speech gesturing (where speech and gesture are processed as a single, fluid communicative act) to make the interaction feel even more human-like. * Releasing parts of the dataset to the public to foster further research in assistive HRI.

Summary updated

Abstract

We explore new aspects of assistive living on smart human-robot interaction (HRI) that involve automatic recognition and online validation of speech and gestures in a natural interface, providing social features for HRI. We introduce a whole framework and resources of a real-life scenario for elderly subjects supported by an assistive bathing robot, addressing health and hygiene care issues. We contribute a new dataset and a suite of tools used for data acquisition and a state-of-the-art pipeline for multimodal learning within the framework of the I -Support bathing robot, with emphasis on audio and RGB- D visual streams. We consider privacy issues by evaluating the depth visual stream along with the RGB, using Kinect sensors. The audio-gestural recognition task on this new dataset yields up to 84.5%, while the online validation of the I-Support system on elderly users accomplishes up to 84% when the two modalities are fused together. The results are promising enough to support further research in the area of multimodal recognition for assistive social HRI, considering the difficulties of the specific task.
BibTeX
@inproceedings{zlatintsi2018multimodal,
  author = {Athanasia Zlatintsi and Isidoros Rodomagoulakis and Petros Koutras and A. C. Dometios and Vassilis Pitsikalis and Costas S. Tzafestas and Petros Maragos},
  title = {{Multimodal Signal Processing and Learning Aspects of Human-Robot Interaction for an Assistive Bathing Robot}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=UwKjLzYrXN},
}

Loading related content...

Figure