Gaze-supported Large Language Model Framework for Bi-directional Human-Robot Interaction

Jens V. Rüppel, Andrey Rudenko, Tim Schreiter, Martin Magnusson, Achim J. Lilienthal

Introduction

Traditional human-robot interaction (HRI) systems often rely on rigid, scripted pipelines that excel in repetitive tasks but struggle in unstructured, dynamic environments. In real-world scenarios, human communication is frequently ambiguous—relying on deictic references like "put it in that box" or shifting intentions mid-task. To address these challenges, this research introduces a modular, real-time framework that utilizes a Large Language Model (LLM) as a central reasoning engine. By fusing speech, gaze, and 3D vision, the system enables robots to perceive user intent more naturally and engage in bi-directional communication.

Modular Architecture

The proposed framework is built upon three primary modules designed to facilitate seamless interaction between the human user and the robotic agent.

1. Input Fusion Module

This module serves as the sensory gateway, gathering data from multiple modalities to create a comprehensive understanding of the environment. High-level scene captioning is handled by BLIP-v2, while YOLO-World provides open-vocabulary object detection. To maintain object persistence despite occlusions, the system integrates Segment Anything 2 (SAM 2).

A critical innovation in this module is the integration of Tobii Glasses 3 mobile eye-tracking. By mapping the user's gaze direction to specific objects within a 3D environment, the robot can ground linguistic commands in physical reality. Speech is simultaneously processed via a real-time Whisper implementation, allowing the system to pair verbal instructions with visual focus.

2. Reasoning Module

The fused data—condensed into a structured Python dictionary—is transmitted to GPT-4o-mini. The framework employs Chain-of-Thought (CoT) prompting to guide the LLM through complex task logic. Rather than simply generating natural language responses, the LLM produces executable Python code that calls specific functions within the robot's SDK. This approach allows the robot to perform logical operations, such as counting or sorting, without requiring developers to hard-code every potential scenario.

3. Action Module

The action module serves as the physical interface, executing the LLM-generated scripts. This triggers a variety of responses, including physical movements (e.g., pointing or nodding), synthesized speech, or light signals. The framework's versatility was demonstrated on two distinct platforms: the humanoid NAO robot and the ARMoD (Anthropomorphic Robot Mock Driver) concept for industrial applications.

Experimental Evaluation

The researchers evaluated the framework through both "in the wild" public demonstrations and a controlled lab study involving 29 participants (ages 20–76). The controlled experiment utilized a 12x5m corridor setup, comparing the LLM-based framework against a traditional scripted baseline in tasks involving navigation and object manipulation.

Key Findings

  • Perceived Confidence: Participants reported significantly higher confidence when using the LLM-driven system ($M=5.6$) compared to the scripted baseline ($M=4.4$). The ability of the LLM to provide context-aware hints and vocal feedback helped users feel more supported during the task.
  • Task Engagement: While the LLM condition was praised for its dynamic behavior and clear instructions, it suffered from higher latency due to API inference times. Despite these delays, task execution ratings remained high ($M=6.3$).
  • Resource Consumption: The advanced computational requirements of local vision models and external API calls resulted in the LLM-based approach consuming approximately 10.5% more energy (1972 Wh vs. 1784 Wh) than the scripted alternative.
  • Gaze Grounding: The study confirmed that gaze tracking is an effective mechanism for resolving ambiguity. By seeing what the user is looking at, the robot can resolve "this" or "that" references that would otherwise stall a purely voice-based system.

Limitations and Future Outlook

Despite its success in improving user confidence, the framework faces several hurdles for industrial deployment. The non-deterministic nature of LLMs can lead to "hallucinated" code or unpredictable behavior, which poses safety concerns. Additionally, environmental noise remains a challenge for accurate speech transcription, and the vision system occasionally struggles with small objects or skewed perspectives.

Future work will focus on enhancing scene understanding to allow the robot to describe object properties (such as color, size, or shape) more granularly. By refining the error-handling wrappers and reducing latency, the researchers aim to move closer to a general-purpose robotic "brain" capable of intuitive, bi-directional interaction in any shared workspace.

Summary updated

BibTeX
@inproceedings{ruppel2025gaze,
  author = {Jens V. Rüppel and Andrey Rudenko and Tim Schreiter and Martin Magnusson and Achim J. Lilienthal},
  title = {{Gaze-supported Large Language Model Framework for Bi-directional Human-Robot Interaction}},
  booktitle = {RO-MAN},
  year = {2025},
  doi = {10.1109/ro-man63969.2025.11217612},
}
Figure