TL;DR
Researchers have introduced the first framework and large-scale dataset for gaze following in 360-degree images. By modeling gaze as a 3D sight line rather than a 2D image-plane vector, their "dual-pathway" approach successfully overcomes the massive geometric distortions inherent in spherical projections, significantly outperforming existing 2D-based methods.
The Challenge of 360-Degree Gaze Following
Gaze following—predicting exactly where a person in an image is looking—is a fundamental task for understanding human behavior and social interaction. While well-studied in standard 2D images, it faces two major hurdles in immersive environments:
- The "Out of Frame" Issue: In standard photos, the person is often looking at something outside the camera's limited field of view.
- The Distortion Dilemma: 360-degree images (equirectangular projections) solve the field-of-view problem but introduce heavy distortion. In a normal 2D photo, a person’s line of sight looks like a straight line. In a 360-degree "flat" map, that same straight line of sight becomes a complex curve over long distances.
Existing models designed for 2D images fail in 360-degree environments because they assume gaze follows a straight 2D path, leading to massive errors when a subject is looking at a distant object across the "sphere."
Methods: A 3D Sight Line Guided Dual-Pathway Framework
The authors propose a framework consisting of three main modules to address these geometric challenges:
- Gaze Direction Estimation (GDE): Instead of predicting a 2D arrow, this module uses a ResNet-50 backbone to estimate a 3D gaze vector in spherical coordinates. Crucially, it accounts for the subject's position on the sphere to handle local distortion.
- Dual-Pathway Prediction (DP): This is the core innovation. It splits the search into two regions:
- The Distant Pathway ("There"): It projects the 3D sight line onto the sphere and finds where it intersects the environment. It then crops a region around this intersection to look for the target.
- The Local Pathway ("Here"): For short-distance gaze (where distortion is minimal), it searches a 2D cone-shaped field near the subject's head.
- Dual-Pathway Fusion (DF): A final convolutional layer fuses the heatmaps from both pathways to determine the most likely gaze target.
The GazeFollow360 Dataset
The paper introduces GazeFollow360, the first large-scale dataset for this task, featuring ~10,000 high-resolution (4K) 360-degree images. The content includes diverse indoor and outdoor scenes crawled from YouTube.
Annotations include bounding boxes for heads and precise gaze target coordinates, verified by multiple human annotators using specialized 3D viewing software to ensure accuracy despite projection distortions.
Key Results and Insights
The "aha moment" of this research is that gaze following in 360 degrees is primarily a geometry problem rather than a simple image-processing one. Modeling the sight line in 3D sphere space is the only way to maintain accuracy across the entire field of view.
The proposed method achieved a new state-of-the-art for 360-degree gaze following:
- Spherical Distance: The model achieved an error of 0.6067, a significant improvement over the best 2D-adapted model (Zhang et al.), which sat at 0.7366.
- AUC (Area Under Curve): It reached a score of 0.8104, outperforming standard 2D gaze following models like Chong et al. (0.7765).
Visualizations show the model correctly "wrapping" its search around the edges of the equirectangular image, a feat impossible for previous 2D architectures.
Limitations and Future Work
While successful, the system's accuracy is heavily dependent on the initial 3D gaze direction estimate. If the head-pose estimation is slightly off, the 3D sight line might miss the target entirely. Additionally, while GazeFollow360 is the largest of its kind, 10,000 images is still smaller than massive 2D datasets. Future work may involve integrating more advanced 3D gaze estimation algorithms or expanding the dataset to include more dynamic, multi-person social interactions and temporal consistency in video.
Summary updated