Predicting what a human will do next is a fundamental challenge for autonomous systems, from social robots navigating a hallway to AR assistants guiding a repair. Traditional models often rely on "reactive" cues—velocity, position, or current pose—to forecast future movement. However, these signals are inherently ambiguous: a person standing in a kitchen might be about to reach for a cup, open a cupboard, or turn toward the door, all while maintaining the same initial posture.
The key to resolving this ambiguity lies in the eyes. Research across egocentric vision and robotics has established that eye gaze serves as a powerful leading indicator of human intent, typically preceding physical action by hundreds of milliseconds. By treating gaze as a proxy for internal cognitive states, researchers are building systems that don't just react to movement, but anticipate the goals driving it.
The Temporal Lead of Visual Attention
The core intuition behind this research direction is the "eye-hand" or "eye-body" coordination gap. In goal-oriented tasks, humans fixate on a target object or location well before initiating physical interaction. Philipp Kratzer and colleagues, through the MoGaze dataset, demonstrated that this anticipatory signal allows models to predict manipulation actions approximately 0.5 seconds before they begin.
This temporal gap is not just a quirk of human biology; it is a functional requirement for motor planning. As explored in A POMDP Model of Eye-Hand Coordination, the brain plans information-gathering sequences several steps ahead. Consequently, tracking where a user looks provides a "pre-signal" that narrows the search space for potential future actions Zhong et al..
Disambiguating Future Paths
When two different intentions result in the same initial physical trajectory, motion-only models fail. For instance, a person walking toward a table might intend to sit down or simply pass by it. Yanchao Yang and the creators of the GIMO framework show that 3D eye gaze acts as the "tie-breaker." By identifying the specific interaction target (the chair vs. the space beyond the table) before physical movement begins, gaze-informed models can accurately predict long-term destination-specific body configurations.
This predictive power extends to social navigation. Justin W. Hart and Peter Stone found that integrating gaze orientation with spatial position allows robots to infer a human's intended destination much earlier than models relying on velocity alone, facilitating smoother, more "human-like" hallway passing Hart et al..
Multimodal Fusion and Latent Intentions
While gaze is a potent signal, it is also "noisy"—our eyes frequently dart around (saccades) for reasons unrelated to our primary goal. To handle this, researchers use multimodal fusion and hierarchical modeling.
- Vislets and Tracklets: Irtiza Hasan and Marco Cristani introduced the MX-LSTM, which mixes "tracklets" (positional data) with "vislets" (short head-pose sequences). This helps resolve ambiguity in low-velocity scenarios where traditional movement cues are silent.
- Latent Goal Inference: Rather than mapping raw gaze points directly to actions, Ping Wei and Song-Chun Zhu propose modeling "latent intentions"—hidden goals like "locating an object"—as the drivers of gaze. This hierarchical approach Wei et al. smooths out jittery sensor data and improves prediction by coupling attention and intention.
- Semantic Labeling: In assistive robotics, Henny Admoni and Reuben M. Aronson use a two-stage pipeline that first labels fixations on environmental objects before using a Hidden Markov Model to infer the user's goal Admoni et al..
Applications in Proactive Systems
The shift from reactive to proactive AI is most visible in three domains:
- Assistive AI: Datasets like HoloAssist and HARMONIC enable assistants to provide guidance before a user makes a mistake. If the system predicts a gaze trajectory that deviates from the "expert" path, it can flag a potential error Furnari et al..
- Autonomous Driving: Models like RouteFormer leverage driver field-of-view to predict complex maneuvers like roundabout exits or sharp turns, where steering input alone arrives too late for safe coordination Savov.
- Implicit Interfaces: In XR, "implicit gaze" research Sendhilnathan et al. moves away from using eyes as a cursor. Instead, the system monitors subconscious eye movements to pre-activate or highlight UI elements before the user's hand even arrives, reducing interaction friction.
By treating the eyes as a window into the immediate future, this research direction transforms passive observation into proactive partnership, allowing machines to align with human intent before a single step is taken.