Gaze as a Leading Indicator for Intent Prediction

This direction is split into other directions: gaze-based action anticipation. Its last version stays readable and citable here.

Version 1 of 1198 papers305 conceptsfrozen 2026-10-04

Predicting what a human will do next is a fundamental challenge for autonomous systems, from social robots navigating a hallway to AR assistants guiding a repair. Traditional models often rely on "reactive" cues—velocity, position, or current pose—to forecast future movement. However, these signals are inherently ambiguous: a person standing in a kitchen might be about to reach for a cup, open a cupboard, or turn toward the door, all while maintaining the same initial posture.

The key to resolving this ambiguity lies in the eyes. Research across egocentric vision and robotics has established that eye gaze serves as a powerful leading indicator of human intent, typically preceding physical action by hundreds of milliseconds. By treating gaze as a proxy for internal cognitive states, researchers are building systems that don't just react to movement, but anticipate the goals driving it.

The Temporal Lead of Visual Attention

The core intuition behind this research direction is the "eye-hand" or "eye-body" coordination gap. In goal-oriented tasks, humans fixate on a target object or location well before initiating physical interaction. Philipp Kratzer and colleagues, through the MoGaze dataset, demonstrated that this anticipatory signal allows models to predict manipulation actions approximately 0.5 seconds before they begin.

This temporal gap is not just a quirk of human biology; it is a functional requirement for motor planning. As explored in A POMDP Model of Eye-Hand Coordination, the brain plans information-gathering sequences several steps ahead. Consequently, tracking where a user looks provides a "pre-signal" that narrows the search space for potential future actions Zhong et al..

Disambiguating Future Paths

When two different intentions result in the same initial physical trajectory, motion-only models fail. For instance, a person walking toward a table might intend to sit down or simply pass by it. Yanchao Yang and the creators of the GIMO framework show that 3D eye gaze acts as the "tie-breaker." By identifying the specific interaction target (the chair vs. the space beyond the table) before physical movement begins, gaze-informed models can accurately predict long-term destination-specific body configurations.

This predictive power extends to social navigation. Justin W. Hart and Peter Stone found that integrating gaze orientation with spatial position allows robots to infer a human's intended destination much earlier than models relying on velocity alone, facilitating smoother, more "human-like" hallway passing Hart et al..

Multimodal Fusion and Latent Intentions

While gaze is a potent signal, it is also "noisy"—our eyes frequently dart around (saccades) for reasons unrelated to our primary goal. To handle this, researchers use multimodal fusion and hierarchical modeling.

  1. Vislets and Tracklets: Irtiza Hasan and Marco Cristani introduced the MX-LSTM, which mixes "tracklets" (positional data) with "vislets" (short head-pose sequences). This helps resolve ambiguity in low-velocity scenarios where traditional movement cues are silent.
  2. Latent Goal Inference: Rather than mapping raw gaze points directly to actions, Ping Wei and Song-Chun Zhu propose modeling "latent intentions"—hidden goals like "locating an object"—as the drivers of gaze. This hierarchical approach Wei et al. smooths out jittery sensor data and improves prediction by coupling attention and intention.
  3. Semantic Labeling: In assistive robotics, Henny Admoni and Reuben M. Aronson use a two-stage pipeline that first labels fixations on environmental objects before using a Hidden Markov Model to infer the user's goal Admoni et al..

Applications in Proactive Systems

The shift from reactive to proactive AI is most visible in three domains:

  • Assistive AI: Datasets like HoloAssist and HARMONIC enable assistants to provide guidance before a user makes a mistake. If the system predicts a gaze trajectory that deviates from the "expert" path, it can flag a potential error Furnari et al..
  • Autonomous Driving: Models like RouteFormer leverage driver field-of-view to predict complex maneuvers like roundabout exits or sharp turns, where steering input alone arrives too late for safe coordination Savov.
  • Implicit Interfaces: In XR, "implicit gaze" research Sendhilnathan et al. moves away from using eyes as a cursor. Instead, the system monitors subconscious eye movements to pre-activate or highlight UI elements before the user's hand even arrives, reducing interaction friction.

By treating the eyes as a window into the immediate future, this research direction transforms passive observation into proactive partnership, allowing machines to align with human intent before a single step is taken.

Papers in this version

Showing 60 of 198 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_gaze_as_a_leading_indicator_for_intent_p_b490a99f,
  title = {Gaze as a Leading Indicator for Intent Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gaze-as-a-leading-indicator-for-intent-prediction/rdr_b490a99fc39b4cfaaf191499c0b184fd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_gaze_as_a_leading_indicator_for_intent_p_b490a99f_v1,
  title = {Gaze as a Leading Indicator for Intent Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gaze-as-a-leading-indicator-for-intent-prediction/rdr_b490a99fc39b4cfaaf191499c0b184fd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}