Depth Aware 3D Gaze Target Detection

31 concepts22 papers1 proposal

Imagine looking through a window at a bird in a distant tree, while a small fly rests on the glass directly in your line of sight. To a standard camera tracking your face, your eyes are pointing along a vector that passes through both the fly and the bird. Because they map to the exact same 2D pixel coordinates, a conventional gaze-tracking system cannot reliably tell which object has captured your attention.

This scenario illustrates the fundamental limitation of traditional gaze tracking: spatial ambiguity along the visual axis. To resolve this, a growing body of research shifts the problem from 2D image planes into 3D space by explicitly integrating scene depth and geometry.

The Ambiguity of the 2D Visual Axis

When we observe someone looking at an environment, a standard 2D image captures their head orientation and eye direction as flat projections. However, as noted by Soma Nonaka et al., estimating gaze solely on a 2D image plane prevents a true understanding of where a person is looking within a three-dimensional physical space.

The core issue is geometric. A 2D gaze vector drawn from an observer's eye extends infinitely as a ray. Any number of objects can sit along this line of sight. As Yuxue Yang et al. point out, standard monocular imagery suffers from a loss of depth information, making it exceptionally difficult to transition from 2D pixels to precise 3D coordinates. Without depth, a model cannot distinguish between a foreground target and a background element that happen to align perfectly from the camera's perspective (Francesco Tonini et al.). As a result, 2D models are easily distracted by visually salient objects that are actually completely irrelevant to the observer's actual focus (Feng Lu et al.).

2D Perspective (Ambiguous):
[Eye] ----------> (Foreground Object / Distractor) ----------> (Background Target)
                  ^ Both appear at the same 2D pixel coordinate!

The Mechanism: Filtering with 3D Gaze Cones

To resolve this spatial ambiguity, modern architectures transition from predicting flat 2D heatmaps to modeling gaze as a 3D spatial intersection. The fundamental mechanism relies on combining directional gaze data with scene depth data, which can be inferred via monocular depth estimation or captured directly through RGBD sensors.

Instead of treating gaze as an infinitely thin line, Yi Fang et al. and Francesco Tonini et al. propose modeling human attention as a 3D gaze cone or volume. This cone accounts for the natural angular uncertainty of human vision. By calculating the intersection of this directional cone with a depth map of the scene, the system generates a 3D field-of-view heatmap (Samy Tafasca et al.).

3D Volumetric Filtering:
                /--- (Object A: Depth = 1m) -> Out of focus range
[Eye] === Gaze Cone ===> [Intersection Zone: Depth = 3m] -> Target Located!
                \--- (Object B: Depth = 5m) -> Out of focus range

This intersection acts as a powerful geometric filter. Depth information serves as a critical disambiguator by ruling out salient objects that are physically distant from the observer or blocked by foreground geometry (Anshul Gupta et al.). If an object lies along the 2D gaze path but its physical depth does not align with the intersection of the gaze cone and the scene's geometry, it is filtered out.

Volumetric Search and Occlusion Handling

Why does this volumetric approach perform better than traditional 2D scanning windows? The answer lies in physical scale and occlusion handling. In a traditional 2D search, background clutter and foreground targets are mixed together if they appear adjacent on the image plane.

As demonstrated in foundational depth-based tracking work by the authors of Depth-Based Hand Pose Estimation: Data, Methods, and Challenges, a 3D volumetric search mechanism inherently accounts for physical scale. This allows a system to ignore background clutter that is physically distant from the target, even when that clutter appears adjacent in a 2D projection.

This geometric constraint is highly effective during occlusions. For instance, Bowei Zhang et al. utilize 3D spatial proximity to define search windows, preventing tracking errors when foreground and background objects overlap in the image plane. Similarly, in human-object interaction tasks, Xiaoyang Liu et al. show that integrating 3D spatial reconstruction with monocular depth estimation resolves view ambiguity by filtering out distractor objects that appear proximal in a 2D plane but are physically distant in 3D space.

From Vectors to Vergence and Surfaces

When depth is fully integrated into the architecture, gaze tracking evolves from simple directional alignment to true spatial localization. This manifests in two primary ways:

  • Surface Projection: By leveraging high-resolution 3D point clouds, a model can project a gaze vector directly onto the nearest physical surface (Meng Liu et al.). This method significantly reduces depth estimation errors compared to traditional binocular triangulation, overcoming the accuracy degradation typically seen when tracking gaze on objects several meters away.
  • Gaze Vergence: Beyond projecting rays onto surfaces, understanding the internal anatomy of the eye allows models to estimate the exact depth at which the eyes converge. Yiwei Bao demonstrates that incorporating 3D structural annotations of the eyeball enables models to estimate gaze vergence, allowing user interfaces to respond directly to the depth of a user's focus within a 3D environment.

By mapping these 3D trajectories, systems can perform complex downstream tasks entirely through gaze. For example, Daniel Weber et al. demonstrate that transforming gaze trajectories into 3D heatmaps allows for highly accurate object detection and bounding-box estimation without relying on traditional image-based object detectors. Likewise, mapping a user's 2D gaze coordinates onto a 3D depth map allows systems to infer spatial intent in real time, enabling applications like safe, perception-aware aerial drone teleoperation (Qianhao Wang et al.).

Key Foundational Work

Ultra-low cost eyetracking as an high-information throughput alternative to BMIs (2011) — contributed 1 concept elements to this direction.
Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction (2014) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_depth_aware_spatial_filtering_ae1cd9e8,
  title = {depth-aware spatial filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/depth-aware-spatial-filtering/rdr_ae1cd9e8933b4ff992efd59638857aac},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_depth_aware_spatial_filtering_ae1cd9e8_v1,
  title = {depth-aware spatial filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/depth-aware-spatial-filtering/rdr_ae1cd9e8933b4ff992efd59638857aac/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake