HEIR: Harness Egocentric Intent for Human-Robot Interactions

Yichi Zhang, Bo Wang, Zhenghan Zhu, Yutong Han, Jiayin Zou, Gao Kejun, Chang Liu, Shaoxuan Xie, Chi Zhang, Shi Haoyang, Zongxuan Lai, Sen Cui, Guocai Yao, Miao LiuarXiv 2026 · arXiv

TL;DR

HEIR (Harnessing Egocentric Intent for Human-Robot Interactions) is a new large-scale dataset and benchmark designed to shift embodied AI from being purely task-centric to truly user-centric. By pairing human egocentric cues—like gaze, speech, and video—with robot-side observations and trajectories, it provides a robust foundation for robots to understand and execute complex, long-horizon household tasks that require interleaving navigation and manipulation.

What Problem Does This Paper Address?

Most current research in embodied AI is "task-centric": robots are given explicit, unambiguous instructions to execute, while the human remains peripheral to the problem formulation. However, for a robot to be a useful assistant in a real home, it must be "user-centric."

In naturalistic settings, human requests are often vague or rely on multimodal context—like a glance toward a specific object or a deictic reference ("Bring me that"). Furthermore, real-world tasks are "long-horizon," meaning they aren't just single actions but sequences of navigation and manipulation that evolve over time. Existing datasets often fall short by focusing on short-horizon tabletop interactions or lacking the human's egocentric perspective, which is critical for grounding intent.

Methods

The authors propose a hierarchical execution pipeline to tackle the complexity of user-centric HRI:

  1. Intent Reasoner: A Vision-Language Model (VLM) that integrates human-centric context (egocentric video + gaze), robot observations, and interaction history to infer a structured intent. This includes a grounded task description, target objects, and a "nominal" (idealized) plan.
  2. Closed-loop Planner: A fine-tuned Large Language Model (LLM) that acts as the "brain," deciding at each step whether to Continue the current subtask, Switch to a new one (providing a specific subgoal and completion condition), or mark the episode as Done.
  3. Execution Backends: Specialized models for navigation (Vision-Language Navigation or VLN) and manipulation (Vision-Language-Action or VLA models) that execute the subgoals provided by the planner.
The HEIR real-robot execution pipeline showing how intent is parsed and dispatched to specialized backends.

Datasets

The core contribution is the HEIR dataset, which stands out for its scale and multimodal richness:

  • Scale: 38.56 hours of interaction across 300 sessions, 1,836 episodes, and 6,416 subtasks.
  • Multimodality: It pairs human-side data (egocentric video, eye-tracked gaze, speech) with robot-side data (head/wrist cameras, joint states, and teleoperated trajectories).
  • Complexity: Tasks span 20 categories in naturalistic household environments (kitchen, home office, living room), requiring complex interleaving of moving and grasping.
Statistics and composition of the HEIR dataset, highlighting its long-horizon nature and diverse task categories.

Key Insights and Ideas

  • Gaze is the Ultimate Disambiguator: One of the most striking findings is how much human gaze helps. It allows the robot to resolve ambiguous references ("that thing") and find objects that are currently outside the robot's own field of view but visible to the human.
  • The "Human Perspective" Gap: Models relying solely on robot-view cameras perform significantly worse at understanding intent. Adding the human's egocentric view provides the "missing link" for grounding language in the physical environment.
  • History is a Feature, Not a Bug: In long-horizon sessions, humans often refer back to previous actions. The authors found that modeling interaction history—knowing what the robot did in the last episode—is crucial for maintaining a coherent "memory" and successfully completing subsequent requests.
  • The Brittleness of Integration: While individual components (like navigation) can achieve high success rates, the end-to-end success rate remains low. This highlights a major challenge in the field: error accumulation across the perception-planning-action pipeline.

Key Results

  • Intent Grounding: Using the full human-centric context (Human View + Robot View + Gaze) improved Slot F1 scores to 77.18, compared to just 71.22 when using only the robot's view.
  • Planning: Fine-tuning the closed-loop planner on HEIR data led to massive gains, increasing decision accuracy by 25–37 percentage points over pre-trained baselines.
  • Real-Robot Performance: In physical deployments, the system achieved an 81.25% subtask success rate for navigation. However, the end-to-end episode success rate was only 18.52%, underscoring the difficulty of long-horizon tasks where a single failure in a subtask can derail the entire request.
Figure 1. Example long-horizon HRI session in HEIR. Each recorded session contains successive episodes, each initiated by a natural human request grounded in the user’s egocentric view and gaze. Each request is mapped to a clarified task specification and fulfilled through one or more navigation and manipulation subtasks, while the top-down trajectory visualizes the corresponding spatial execution and state transitions across episodes.

Limitations and Future Work

The authors acknowledge several areas for growth:

  • Error Accumulation: The low end-to-end success rate suggests a need for more robust error recovery and tighter integration between planning and execution.
  • Environment Generalization: While furniture was repositioned to create variety, the data was collected in a single staged room. Future work should explore generalization to entirely unseen floor plans.
  • Autonomous Data Collection: The current dataset relies on teleoperated demonstrations. Moving toward autonomous or semi-autonomous data collection could help scale the resource even further.
  • Real-time Latency: Integrating large VLMs and LLMs into a real-time robot loop remains a computational challenge that requires further optimization.

Summary updated

Abstract

For robots to assist, serve, and collaborate with humans, they must not only interpret human requests conveyed through language and multimodal intentional cues, but also execute long-horizon tasks that interleave navigation and manipulation. In this paper, we introduce HEIR, a resource for studying how to Harness Egocentric Intent for Human-Robot Interaction. HEIR pairs human speech, measured gaze, and egocentric video with robot observations and teleoperated navigation-manipulation trajectories, while each session preserves successive episodes, with each request fulfilled through multiple execution subtasks. The captured data, together with human-authored annotations, support the study of intent grounding, execution planning, and task completion within a unified interaction setting. To demonstrate the value of HEIR, we conduct extensive offline experiments to evaluate how models benefit from additional human-centric context across these tasks, together with real-robot experiments that assess execution task completion. HEIR thus provides a valuable resource for studying user-centric robot interaction in naturalistic household environments. Our demo and dataset are available online.
BibTeX
@misc{zhang2026heir,
  author = {Yichi Zhang and Bo Wang and Zhenghan Zhu and Yutong Han and Jiayin Zou and Gao Kejun and Chang Liu and Shaoxuan Xie and Chi Zhang and Shi Haoyang and Zongxuan Lai and Sen Cui and Guocai Yao and Miao Liu},
  title = {{HEIR: Harness Egocentric Intent for Human-Robot Interactions}},
  year = {2026},
  eprint = {2611.17238},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17238},
}
Figure