For decades, eye-tracking has been treated as a window into the human mind. The underlying assumption is intuitive: where the eyes look, the mind is processing information. In human-computer interaction, robotics, and computer vision, this assumption has driven the integration of gaze data to predict user intent, guide autonomous vehicles, and evaluate learning.
However, raw gaze data is fundamentally deceptive. A growing body of research demonstrates a profound visual-cognitive processing gap—a decoupling between overt gaze (where the eyes point) and covert attention or cognitive intent. Human eyes are erratic, prone to involuntary movements, easily captured by irrelevant visual stimuli, and frequently misaligned with internal mental states. To build systems that truly understand human intent, researchers are shifting from simple gaze tracking to sophisticated models that treat gaze as a noisy, latent signal that must be contextualized and decoded.
The Anatomy of Gaze Noise
To understand why raw gaze data fails as a direct proxy for attention, we must look at the two primary ways gaze dissociates from cognition: environmental/physiological noise and cognitive decoupling.
Physiological and Environmental Distractions
Human eyes do not move in smooth, perfectly rational trajectories. They dart around via rapid, involuntary movements called saccades. Furthermore, our visual system is highly sensitive to bottom-up visual saliency—bright colors, sudden movements, or complex shapes can trigger involuntary fixations that have nothing to do with a user's current goal.
As Yi Liu et al. observe, eye-gaze data is highly susceptible to noise caused by the visual properties of an interface, making it difficult to distinguish between intentional, task-related fixations and involuntary shifts triggered by visual saliency. In real-world environments, this problem compounds. For instance, Anwesan Pal highlights that human drivers are frequently distracted by task-irrelevant stimuli like roadside advertisements or scenery. If an autonomous driving model is trained to mimic raw human gaze without filtering, it risks learning to prioritize features that do not contribute to safe navigation.
Cognitive Decoupling and Mind-Wandering
Even when the eyes are perfectly still, the mind may be elsewhere. "Passive staring" occurs when a user's eyes rest on an object while their mind wanders, a phenomenon that Yuhu Chang et al. note creates significant data pollution for attention-aware wearables.
Conversely, intense cognitive processing often causes humans to look away from the very objects they are thinking about. When reflecting or summarizing information, individuals frequently avert their gaze to neutral or unrelated screen areas to reduce cognitive load. Anam Ahmad Khan et al. demonstrate this decoupling during voice-based note-taking, showing that simple spatial heuristics fail to anchor user annotations because visual fixation completely separates from cognitive focus during complex mental tasks.
[Visual Stimulus] ---> Involuntary Saccade / Saliency Capture ---> Noisy Gaze Point (No Cognitive Intent)
[Internal Thought] --> Gaze Aversion / Passive Staring ---------> Noisy Gaze Point (Misaligned with Focus)
The Peripheral Loophole
Compounding this issue is the fact that humans can process information without looking directly at it. Christian Vater et al. emphasize that peripheral vision is a primary tool for environmental monitoring and action planning; humans routinely navigate hazards and track objects without foveal fixation. Because standard eye-trackers only measure foveal orientation, they completely miss this peripheral processing loop. As André Klostermann et al. argue, this fundamental decoupling means that the absence of a direct gaze point does not imply a lack of awareness, and a functional fixation cannot be distinguished from a passive glance by spatial coordinates alone.
Why Simple Gaze Metrics Fail Downstream
Because of this visual-cognitive gap, downstream applications that rely on raw gaze metrics as ground-truth indicators of attention or performance hit a hard ceiling.
In educational technology, for instance, researchers have attempted to use "gaze synchrony"—the degree to which students look at the same instructional material simultaneously—as a metric for engagement. However, Babette Bühler et al. found that while gaze synchrony confirms that a student is looking at the right place, it fails to predict actual learning performance or knowledge retention. Students can stare at a lecture slide without encoding its content.
This finding is echoed by Feng Tian et al., who show that while eye-tracking can classify reading or watching behaviors, it is an insufficient predictor of cognitive performance on its own. Similarly, in AI-partnered classrooms, Shamya Karumbaiah et al. demonstrate that coarse visual gaze metrics fail to correlate with student learning outcomes because they capture a wide range of non-pedagogical activities, such as behavior management.
The gap is equally pronounced in task performance and expertise. One might assume that guiding a novice's eyes to match an expert's gaze path would accelerate learning. Yet, Brett D. Roads et al. show that while attentional highlighting successfully forces novices to mimic expert gaze patterns, it does not guarantee an improvement in task performance or decision-making accuracy. The cognitive processing of the information at those fixation points remains underdeveloped.
This is because, as Elina Birmingham demonstrates using the moving window technique, correct visual attention to diagnostic features does not automatically lead to accurate categorization if underlying perceptual or cognitive processing deficits are present. Gaze duration simply does not equal cognitive retention or preference, a reality that Francesco Ragusa confirms in behavioral studies where visual dwell time failed to correlate with subsequent memory or subjective preference.
Algorithmic Bridges: Decoding Intent from Noise
To overcome the limitations of raw eye-tracking, recent work has shifted toward architectures that treat gaze not as an answer, but as a noisy feature that must be reconciled with environmental context and task constraints.
Semantic Contextualization
If human gaze is prone to wandering, how can we isolate purposeful fixations? The solution is to filter gaze through a deep understanding of the environment.
In autonomous driving, Anwesan Pal introduced SAGE-Net, a framework that augments raw human gaze data with semantic scene information, depth estimation, and pedestrian intent. By forcing the model to predict where a driver should look for safety rather than where they did look, it filters out task-irrelevant visual distractions.
Similarly, Deepak Gopinath et al. developed the MAAD framework, which combines video data with noisy gaze signals to generate a spatial heatmap of "attended awareness." By cross-referencing gaze points with salient environmental features over time, the model filters out sensor noise and fleeting glances to map what a driver actually perceives and remembers.
Raw Gaze Signal -----\
+---> [Semantic Contextualization Model] ---> Attended Awareness Map
Environmental Video --/ (Filters Saliency & Saccades)
Latent Variable and Intention Modeling
Instead of directly mapping eye movements to system actions—which leads to jittery and unstable control—modern frameworks treat gaze as an indicator of a hidden, latent variable: human intent.
For egocentric activity recognition, Kyle Min et al. modeled human gaze as a structured discrete latent variable within a probabilistic framework. By using a variational method, their model learns to infer task-relevant attention maps, successfully separating informative fixations from background distractors.
In medical robotics, Yuan Bi et al. applied intention estimation to gaze-guided robotic ultrasound systems. Because raw gaze is plagued by non-intentional glances at surrounding context, their architecture applies temporal filtering and intention modeling to ensure the robot only responds to stable, purposeful focus, eliminating jittery motion.
In diagnostic imaging, Anh Nguyen et al. developed RadGazeIntent, a transformer-based model designed to decode the why behind a radiologist's eye movements on chest X-rays. By classifying individual fixations into specific diagnostic intentions, the model successfully isolates goal-directed fixations from transitional or idle eye movements.
Weak Supervision and Prompting Strategies
The misalignment between gaze points and object boundaries also makes gaze a challenging signal for computer vision supervision. Fixations often fall on the center or edges of an object, or bleed into the background.
To use this "cheap" but noisy data for video salient object detection, Qi Qin et al. designed EGCNet. Instead of treating gaze points as exact pixel labels, they use contrastive learning to map noisy fixations to object boundaries, achieving state-of-the-art segmentation without expensive manual annotations.
Finally, to help Multimodal Large Language Models (MLLMs) understand first-person video, Taiying Peng et al. introduced EgoGazeVQA. They demonstrated that standard MLLMs struggle to interpret raw gaze data due to rapid, involuntary saccades. However, by transforming raw gaze into sequential salience heatmaps and using gaze-guided prompting, they enabled the models to filter out the noise, "read between the lines" of human actions, and accurately predict user intent.
Conclusion
The evolution of gaze-aware systems reveals a fundamental truth: looking is not the same as seeing, and seeing is not the same as understanding. Whether due to the physiological realities of saccades, the distractions of a high-saliency environment, or the cognitive tendencies of mind-wandering and reflection, raw gaze data is inherently noisy.
The breakthrough in this field lies in moving away from the naive assumption that gaze equals attention. By treating eye movements as a noisy, latent signal and filtering it through semantic scene context, task constraints, and probabilistic intent modeling, modern architectures successfully bridge the visual-cognitive gap—turning raw sight into actionable insight.
Key Foundational Work
Automated Vocabulary Acquisition and Interpretation in Multimodal Conversational Systems (2007) — contributed 2 concept elements to this direction.
Forming Share Attention betwecn User and Robot Based on Face Posture Estimation (2003) — contributed 1 concept elements to this direction.