Towards Language-Based Modulation of Assistive Robots through Multimodal Models

Philipp Wicke, Lütfi Kerem Senel, Shengqiang Zhang, Luis F. C. Figueredo, Abdeldjallil Naceri, Sami Haddadin, Hinrich SchützearXiv 2023 · DOI · arXiv

Introduction

In the emerging field of Geriatronics—robotics specifically designed for elderly care—the ability for robots to understand and adapt to human instructions in real-time is paramount. Current Human-Robot Interaction (HRI) models often rely on "one-shot" communication, where a robot attempts to execute a command without further input. However, real-world environments are inherently dynamic and ambiguous. Researchers from LMU Munich and TU Munich, in a paper presented at the Geriatronics Summit 2023, propose a novel framework to bridge this gap through mid-task language-based modulation.

The Challenge: Beyond One-Shot Commands

Most existing robotic datasets focus on static instructions or high-level planning. In a typical scenario, a user might say "Bring the medication," but if multiple bottles are present, the robot may choose the wrong one. Without a mechanism for mid-task correction, the user must wait for the failure to occur before re-issuing a command. This lack of a "feedback loop" creates a significant barrier to trust and efficiency in assistive settings. Furthermore, robots often struggle with the "semantic gap"—grounding descriptive adjectives like "the small blue box" into internal visual representations.

Proposed Framework: Language-Based Modulation

The researchers introduce the concept of Language-Based Modulation, which allows users to refine or correct a robot's actions while they are in progress. They categorize these modulations into two types:

  1. Low-Level (LL) Modulation: Modifying specific parameters of an action, such as "be more gentle" or "move slower."
  2. High-Level (HL) Modulation: Altering the sequence or nature of sub-tasks, such as "avoid the carpet" or "pick up the other cup first."
Figure 3: An overview of the modulation process. At Timestep 1, the robot receives an initial command. At Timestep m, a mid-task modulation ('Stack object 2 first!') is provided, altering the robot's trajectory by Timestep t.

Transparent Shared Representations

To address the ambiguity of natural language, the authors propose using Transparent Shared Representations. Instead of relying solely on the robot's ability to parse complex descriptors, the system provides a real-time video feed to the user where objects are highlighted with synthetic labels (e.g., "obj1", "obj2").

Figure 4: Synthetic labels provide a 'common ground' between the human and the robot, allowing for unambiguous referencing of objects in the scene.

By using these specific identifiers, the user can bypass the robot's potential failure to understand descriptive language. This approach treats the human as an active participant in the robot's perception loop, increasing system reliability through transparency.

Technical Implementation

The architecture utilizes a multimodal input encoder that processes both camera feeds and text instructions. A transformer-based policy then predicts the robot's next actions, including position, orientation, and gripper state. To help the model learn these associations, a novel "augmentation module" replaces specific object names in the training text with a generic TARGET token, which the model learns to ground to the highlighted visual object.

Dataset and Preliminary Results

The project extends the RLBench benchmark, a standard for robot learning. The authors are crowdsourcing a new dataset containing 50 natural language instructions for each of 30 household tasks, covering various levels of modulation and instruction specificity.

As a proof of concept, the researchers demonstrated the framework on a "stack the cups" task. They successfully implemented a modulation instruction—"stack the other cup first"—which correctly altered the robot's planned sequence in real-time. This validates the feasibility of using synthetic labels to resolve reference ambiguities during execution.

Conclusion and Future Work

While this research is in its early stages, it establishes a foundational architecture for more interactive and transparent assistive robots. Future work will focus on scaling the model to a wider variety of tasks and evaluating the user experience. A key question remains whether elderly users will find the synthetic label interface intuitive or if more naturalistic interaction methods are required. Nonetheless, treating modulation as a "delta" to an existing trajectory offers a computationally efficient and robust path forward for interactive robotics in eldercare.

Summary updated

BibTeX
@misc{wicke2023towards,
  author = {Philipp Wicke and Lütfi Kerem Senel and Shengqiang Zhang and Luis F. C. Figueredo and Abdeldjallil Naceri and Sami Haddadin and Hinrich Schütze},
  title = {{Towards Language-Based Modulation of Assistive Robots through Multimodal Models}},
  year = {2023},
  eprint = {2306.14830},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.14830},
}
Figure