Depth-Aware Gaze Following and 3D Interaction Grounding

This direction is split into other directions: depth-aware spatial filtering. Its last version stays readable and citable here.

Version 1 of 124 papers38 conceptsfrozen 2026-10-04

In computer vision, understanding where a person is looking—gaze following—has traditionally been treated as a 2D problem. We draw a vector from a person's eyes and look for an intersection with an object in the image plane. However, the real world is three-dimensional, and 2D projections create "perspective traps." Two objects might appear perfectly aligned along a 2D gaze vector, yet one could be inches from the observer while the other is miles away.

Recent research has shifted toward 3D interaction grounding, a paradigm that integrates monocular depth estimation and geometric priors to resolve spatial ambiguity. By treating gaze as a volumetric cone rather than a flat arrow, models can distinguish between what is visually proximal and what is physically relevant.

The 2D Perspective Trap

The fundamental challenge in gaze estimation is that a single 2D image lacks explicit depth. As Soma Nonaka, Ko Nishino, and Shohei Nobuhara argue in Dynamic 3D Gaze from Afar, estimating gaze solely on a 2D plane prevents a true understanding of a person's intent in physical space. Without a 3D vector, it is impossible to map gaze to specific objects in complex, cluttered environments.

This ambiguity is particularly problematic in Human-Object Interaction (HOI) detection. In a flat image, a person’s hand might appear to be touching a distant car, simply because they overlap in the 2D projection. Jian Yang and colleagues address this in HORP: Human-Object Relation Priors Guided HOI Detection by using 3D location priors. By estimating relative depth and spatial occupancy, their model filters out "hallucinated" interactions that are physically impossible in 3D space.

From Vectors to 3D Gaze Cones

To solve this, researchers have introduced the concept of the 3D Gaze Cone. Instead of a line, the model projects a volume into a depth-mapped scene.

Yi Fang and Wei Shen developed a "Dual Attention Module" in Dual Attention Guided Gaze Target Detection in the Wild that calculates the intersection of a directional cone and a monocular depth map. This allows the system to ignore salient objects that are visually aligned in 2D but separated by significant physical distance.

Similarly, Francesco Tonini and Elisa Ricci demonstrate in Object-aware Gaze Target Detection that a geometric gaze cone acts as a spatial filter. It constrains the search space to the person's actual field of view, resolving depth-based ambiguities that 2D heatmaps cannot handle.

Disambiguation through Physical Consistency

The intuition behind these models is physical consistency. If we know the 3D structure of a scene, we can ask: Is this object reachable? Is it occluded?

In A Modular Multimodal Architecture for Gaze Target Prediction, Anshul Gupta and Jean-Marc Odobez show that integrating 3D depth allows a model to rule out distant objects that might be visually distracting but are physically irrelevant to the observer. This is especially powerful in privacy-sensitive settings; the model can predict gaze targets using body pose and depth even if the person's face is blurred.

This 3D awareness also explains why humans are so much better at gaze estimation in person than on video. Daniel Harari and Shimon Ullman found that stereoscopic depth cues—not just motion—are the primary reason for our high accuracy in live social interactions (Measuring and modeling the perception of natural and unconstrained gaze).

Applications in Dynamic Environments

The shift toward 3D grounding is transforming several specialized fields:

  • Autonomous Driving: In World Model-Based End-to-End Scene Generation, Yanchen Guan and colleagues use 3D depth to prevent "perspective traps" where distant hazards appear close on a dashboard camera. Calculating Euclidean distances in 3D space rather than pixel distances is critical for accurate accident anticipation.
  • Developmental Psychology: Samy Tafasca and Anshul Gupta introduced the ChildPlay benchmark, which uses 3D geometry to track children's gaze. They found that 3D field-of-view heatmaps are essential for distinguishing between objects a child is playing with and background clutter.
  • Wearable Computing: Systems like VergeIO are moving beyond 2D coordinates to sense "vergence"—the inward movement of eyes when focusing on near objects—allowing smart glasses to understand the depth of a user's focus.

The Latent 3D Logic of 2D Screens

Perhaps most surprisingly, 3D structure influences how we look at flat screens. Learning Gaze Transitions from Depth to Improve Video Saliency Estimation demonstrates that even when watching 2D video, human gaze patterns are driven by the underlying 3D geometry of the scene. By training models to recognize these latent depth cues, researchers achieved a 15% improvement in predicting where people look on standard displays.

By moving from "where on the image" to "where in the room," this research direction is bringing machine perception closer to the spatial logic of human vision. The goal is no longer just to find a coordinate, but to ground human attention within the physical constraints of the three-dimensional world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_depth_aware_gaze_following_and_3d_intera_026de49b,
  title = {Depth-Aware Gaze Following and 3D Interaction Grounding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/depth-aware-gaze-following-and-3d-interaction-grounding/rdr_026de49b75dc4a5d8ba95c2c9ce3a38b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_depth_aware_gaze_following_and_3d_intera_026de49b_v1,
  title = {Depth-Aware Gaze Following and 3D Interaction Grounding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/depth-aware-gaze-following-and-3d-interaction-grounding/rdr_026de49b75dc4a5d8ba95c2c9ce3a38b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}