Looking here or there? Gaze Following in 360-Degree Images

Yunhao Li, Wei Shen, Zhongpai Gao, Yucheng Zhu, Guangtao Zhai, Guodong Guo

TL;DR

Researchers have introduced the first framework and large-scale dataset for gaze following in 360-degree images. By modeling gaze as a 3D sight line rather than a 2D image-plane vector, their "dual-pathway" approach successfully overcomes the massive geometric distortions inherent in spherical projections, significantly outperforming existing 2D-based methods.

The Challenge of 360-Degree Gaze Following

Gaze following—predicting exactly where a person in an image is looking—is a fundamental task for understanding human behavior and social interaction. While well-studied in standard 2D images, it faces two major hurdles in immersive environments:

  1. The "Out of Frame" Issue: In standard photos, the person is often looking at something outside the camera's limited field of view.
  2. The Distortion Dilemma: 360-degree images (equirectangular projections) solve the field-of-view problem but introduce heavy distortion. In a normal 2D photo, a person’s line of sight looks like a straight line. In a 360-degree "flat" map, that same straight line of sight becomes a complex curve over long distances.
Comparison of gaze following in standard 2D images versus 360-degree equirectangular projections.

Existing models designed for 2D images fail in 360-degree environments because they assume gaze follows a straight 2D path, leading to massive errors when a subject is looking at a distant object across the "sphere."

Methods: A 3D Sight Line Guided Dual-Pathway Framework

The authors propose a framework consisting of three main modules to address these geometric challenges:

  • Gaze Direction Estimation (GDE): Instead of predicting a 2D arrow, this module uses a ResNet-50 backbone to estimate a 3D gaze vector in spherical coordinates. Crucially, it accounts for the subject's position on the sphere to handle local distortion.
  • Dual-Pathway Prediction (DP): This is the core innovation. It splits the search into two regions:
  • The Distant Pathway ("There"): It projects the 3D sight line onto the sphere and finds where it intersects the environment. It then crops a region around this intersection to look for the target.
  • The Local Pathway ("Here"): For short-distance gaze (where distortion is minimal), it searches a 2D cone-shaped field near the subject's head.
  • Dual-Pathway Fusion (DF): A final convolutional layer fuses the heatmaps from both pathways to determine the most likely gaze target.

The GazeFollow360 Dataset

The paper introduces GazeFollow360, the first large-scale dataset for this task, featuring ~10,000 high-resolution (4K) 360-degree images. The content includes diverse indoor and outdoor scenes crawled from YouTube.

Statistical distribution of gaze targets and directions in the GazeFollow360 dataset across longitude, latitude, yaw, and pitch.

Annotations include bounding boxes for heads and precise gaze target coordinates, verified by multiple human annotators using specialized 3D viewing software to ensure accuracy despite projection distortions.

Key Results and Insights

The "aha moment" of this research is that gaze following in 360 degrees is primarily a geometry problem rather than a simple image-processing one. Modeling the sight line in 3D sphere space is the only way to maintain accuracy across the entire field of view.

The proposed method achieved a new state-of-the-art for 360-degree gaze following:

  • Spherical Distance: The model achieved an error of 0.6067, a significant improvement over the best 2D-adapted model (Zhang et al.), which sat at 0.7366.
  • AUC (Area Under Curve): It reached a score of 0.8104, outperforming standard 2D gaze following models like Chong et al. (0.7765).

Visualizations show the model correctly "wrapping" its search around the edges of the equirectangular image, a feat impossible for previous 2D architectures.

Limitations and Future Work

While successful, the system's accuracy is heavily dependent on the initial 3D gaze direction estimate. If the head-pose estimation is slightly off, the 3D sight line might miss the target entirely. Additionally, while GazeFollow360 is the largest of its kind, 10,000 images is still smaller than massive 2D datasets. Future work may involve integrating more advanced 3D gaze estimation algorithms or expanding the dataset to include more dynamic, multi-person social interactions and temporal consistency in video.

Summary updated

Abstract

Gaze following, i.e., detecting the gaze target of a human subject, in 2D images has become an active topic in computer vision. However, it usually suffers from the out of frame issue due to the limited field-of-view (FoV) of 2D images. In this paper, we introduce a novel task, gaze following in 360-degree images which provide an omnidirectional FoV and can alleviate the out of frame issue. We collect the first dataset, "GazeFollow360" <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> , for this task, containing around 10,000 360-degree images with complex gaze behaviors under various scenes. Existing 2D gaze following methods suffer from performance degradation in 360degree images since they may use the assumption that a gaze target is in the 2D gaze sight line. However, this assumption is no longer true for long-distance gaze behaviors in 360-degree images, due to the distortion brought by sphere-to-plane projection. To address this challenge, we propose a 3D sight line guided dual-pathway framework, to detect the gaze target within a local region (here) and from a distant region (there), parallelly. Specifically, the local region is obtained as a 2D cone-shaped field along the 2D projection of the sight line starting at the human subject’s head position, and the distant region is obtained by searching along the sight line in 3D sphere space. Finally, the location of the gaze target is determined by fusing the estimations from both the local region and the distant region. Experimental results show that our method achieves significant improvements over previous 2D gaze following methods on our GazeFollow360 dataset.
BibTeX
@inproceedings{li2021looking,
  author = {Yunhao Li and Wei Shen and Zhongpai Gao and Yucheng Zhu and Guangtao Zhai and Guodong Guo},
  title = {{Looking here or there? Gaze Following in 360-Degree Images}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00372},
}
Figure