Mindeye-Omniassist: A Gaze-Driven LLM-Enhanced Assistive Robot System for Implicit Intention Recognition and Task Execution

Zhisen Zhang, Bo Yang, Xinxing Chen, Weimei Shi, Haoyuan Wang, Wei Luo, Jian Huang2025 · DOI

TL;DR

This paper introduces MindEye-OmniAssist, a novel gaze-driven assistive robot system that leverages Large Language Models (LLMs) and Vision Foundation Models (VFMs) to recognize complex, implicit user intentions beyond predefined tasks. By integrating gaze input with these powerful models, the system can infer user goals and generate appropriate action sequences, paving the way for more versatile and user-friendly human-robot interaction in assistive robotics.

What Problem Does This Paper Address?

Current gaze-based assistive robotic systems are often limited to basic grasping actions and struggle with recognizing a wide range of user intentions. This limitation stems from a reliance on pre-defined tasks and domain-specific prior knowledge, hindering their ability to adapt to diverse and undefined scenarios. The core challenge is to move beyond simple, explicit commands and enable robots to understand subtle, implicit user goals through natural gaze behavior, thereby offering more comprehensive assistance.

Methods

The MindEye-OmniAssist system employs a three-pronged approach:

  1. Open Vocabulary Object Detection (VFM-powered): It utilizes the YOLO-World model, a Vision Foundation Model, to identify objects in the scene. This is crucial because it can detect objects not explicitly programmed into the system.
  2. Gaze-Based Implicit Intention Recognition Network: This network takes object detection results and gaze points as input. It calculates features like the distance from the gaze point to an object's center and the ratio of distances to determine if a user intends to interact with an object. This network uses multi-scale convolutional layers, positional encoders, and Transformer encoders to process sequential gaze data.
  3. LLM-Powered Intention Inference and Action Planning: Once potential interaction objects are identified, their labels are fed into an LLM. The LLM, leveraging its vast knowledge base, infers the user's complete intention (e.g., "brush teeth" or "clean cup") and generates a sequence of actions. The system also includes an intention confirmation module where users can verbally or via gaze signal agree or reject the inferred intention.

Datasets

The paper introduces a new eye movement dataset collected using the Pupil Invisible eye tracker. This dataset comprises data from eight volunteers performing various tasks, including gazing at objects to express interaction intentions and simulating natural eye movements. The dataset includes scene images, 2D gaze point coordinates, and manually annotated labels for interaction intent.

Key Insights and Ideas

  • LLMs as the "Brain" for Implicit Intent: The paper demonstrates that LLMs are powerful tools for inferring complex, multi-object intentions from gaze cues, moving beyond simple object-action mappings.
  • Open Vocabulary is Key: By using VFMs for object detection, the system can handle a much wider array of objects and tasks than systems relying on fixed object lists.
  • Gaze as a Rich Signal: The research highlights how subtle gaze patterns, combined with object context, can convey rich implicit information about user goals.
  • User Confirmation Loop: The inclusion of an explicit confirmation step (via voice or gaze) is vital for building trust and ensuring the robot acts on the correct inferred intention.

Key Results

The MindEye-OmniAssist system achieved promising results in real-world experiments:

  • Overall Success Rate: 41 out of 55 tasks were successfully completed across various undefined tasks.
  • Intention Recognition Accuracy: The proposed gaze-based intention recognition network achieved an experimental-based accuracy of 96.52 ± 1.69%, outperforming existing methods like Midas-Net (91.13 ± 2.44%) and a fixation-based method (61.17 ± 2.93%). Subject-based evaluation showed an average accuracy of 93.63 ± 1.39%.
  • System Evaluation: The system demonstrated success rates for intention recognition (52/55), action planning (50/52), and action execution (41/50) in a desktop environment.

Limitations and Future Work

  • Limitations: The paper acknowledges that the system's performance is dependent on the quality of gaze data and the LLM's understanding. While the success rate is encouraging, it's not perfect, and failures were recorded, often due to issues in intention recognition, action planning, or execution. The current system relies on a predefined set of APIs for action execution.
  • Future Work: The authors plan to focus on improving the accuracy of action planning, developing more effective methods for updating the basic action library, and implementing error correction mechanisms. They also aim to explore more complex and diverse tasks, further enhancing the system's versatility and user-friendliness.

Summary updated

Abstract

A promising effective human-robot interaction in assistive robotic systems is gaze-based control. However, current gaze-based assistive systems mainly help users with basic grasping actions, offering limited support. Moreover, the restricted intent recognition capability constrains the assistive system's ability to provide diverse assistance functions. In this paper, we propose an open implicit intention recognition framework powered by Large Language Model (LLM) and Vision Foundation Model (VFM), which can process gaze input and recognize user intents that are not confined to predefined or specific scenarios. Furthermore, we implement a gaze-driven LLM-enhanced assistive robot system (MindEye-OmniAssist) that recognizes user's intentions through gaze and assists in completing task. To achieve this, the system utilizes open vocabulary object detector, intention recognition network and LLM to infer their full intentions. By integrating eye movement feedback and LLM, it generates action sequences to assist the user in completing tasks. Real-world experiments have been conducted for assistive tasks, and the system achieved an overall success rate of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$41 / 55$</tex> across various undefined tasks. Preliminary results show that the proposed method holds the potential to provide a more user-friendly humancomputer interaction interface and significantly enhance the versatility and effectiveness of assistive systems by supporting more complex and diverse task.
BibTeX
@misc{zhang2025mindeye,
  author = {Zhisen Zhang and Bo Yang and Xinxing Chen and Weimei Shi and Haoyuan Wang and Wei Luo and Jian Huang},
  title = {{Mindeye-Omniassist: A Gaze-Driven LLM-Enhanced Assistive Robot System for Implicit Intention Recognition and Task Execution}},
  year = {2025},
  doi = {10.1109/cbs65871.2025.11267731},
}

Loading related content...

Figure