In computer vision, two tasks are often treated in isolation: estimating where a person is looking (gaze estimation) and identifying what is in a scene (object detection). However, human vision is inherently object-centric. We rarely stare at empty space; we look at things.
A growing body of research suggests that these two tasks are most effective when they constrain one another. By treating scene content as a spatial prior for gaze, and gaze as a semantic filter for objects, researchers have developed systems that are more robust than the sum of their parts.
Saliency as a Gaze Anchor
Raw gaze estimation is notoriously noisy. Whether due to low-cost hardware or rapid eye movements, a predicted gaze vector often points to a "meaningless" patch of pixels. The core intuition of this research direction is that if a gaze estimate falls near a highly salient object, the estimate is likely slightly off, and the true focus is the object itself.
Nicu Sebe, Roberto Valenti, and Theo Gevers formalized this by integrating scene saliency as a prior probability map in [What Are You Looking at?
- Improving Visual Gaze Estimation by Saliency](/work/what-are-you-looking-at-improving-visual-gaze). By shifting raw gaze coordinates toward local saliency maxima, they effectively "snapped" noisy predictions to the most likely visual features.
This "snap-to-object" mechanism serves as a powerful regularizer. As Adrià Recasens and Antonio Torralba demonstrated in Following Gaze Across Views, using object saliency as a spatial prior prevents models from predicting fixations on empty or uninformative regions, even when the person and the object are in different camera views.
Gaze as a Semantic Filter
The relationship also works in reverse: gaze data can be used to prune the "noise" of object detection. Standard detectors often produce numerous false positives, especially in cluttered environments. However, if a human observer is looking at a specific region, the objects in that region are more likely to be semantically relevant.
Kiwon Yun and Dimitris Samaras explored this in Studying Relationships between Human Gaze, Description, and Computer Vision. They used eye-tracking data as a "relevance filter" to discard candidate bounding boxes that did not overlap with human fixations. This approach doesn't just improve precision; it helps the system identify which objects are "important" enough to be described or interacted with.
S. Karthikeyan took this further in From Where and How to What We See, showing that human gaze patterns are so distinct that they can act as a "semantic prior" to help detect challenging categories like text or small animals. By narrowing the search space to fixated regions, detectors can achieve higher accuracy with lower computational cost.
Grounding Attention through Spatial Overlap
To bridge these two signals, researchers often use geometric fusion. The most common mechanism is calculating the intersection between a gaze probability distribution (a heatmap) and object proposals (bounding boxes).
In the context of robotics, Shiva Hanifi and Maria Lombardi used this in iCub Detecting Gazed Objects: A Pipeline Estimating Human Attention. By calculating the Intersection over Union (IoU) between the peak of a gaze heatmap and detected object boxes, the iCub robot can identify exactly which toy a human partner is attending to.
A similar logic applies to social scene understanding. Lifeng Fan and Wenguan Wang showed in Inferring Shared Attention in Social Scene Videos that masking gaze heatmaps with object proposals effectively filters out "accidental" gaze intersections. If two people's gaze lines cross in empty space, the system ignores it; if they cross on an object, it identifies "shared attention."
Beyond Location: Intent and Context
The integration of gaze and objects moves computer vision from simple localization toward understanding intent. In Interact as You Intend, Bingjie Xu and Mohan Kankanhalli used gaze-driven contextual attention to disambiguate human-object interactions. Knowing that a person is looking at a cup while their hand moves toward it provides the "semantic gold" needed to distinguish "picking up a cup" from "reaching across the table."
This synergy is now being used to "warm-start" complex tasks. Nóra Horányi and Hyung Jin Chang demonstrated in G-DAIC that initializing image cropping searches at gaze points can reduce runtime by over 90%. By assuming the user's focus is the most important part of the image, the system bypasses the need to evaluate thousands of irrelevant candidate crops.