Boosting Image-based Mutual Gaze Detection using Pseudo 3D Gaze

Bardia Doosti, Ching-Hui Chen, Raviteja Vemulapalli, Xuhui Jia, Yukun Zhu, Bradley Green

Abstract

Mutual gaze detection, i.e., predicting whether or not two people are looking at each other, plays an important role in understanding human interactions. In this work, we focus on the task of image-based mutual gaze detection, and propose a simple and effective approach to boost the performance by using an auxiliary 3D gaze estimation task during the training phase. We achieve the performance boost without additional labeling cost by training the 3D gaze estimation branch using pseudo 3D gaze labels deduced from mutual gaze labels. By sharing the head image encoder between the 3D gaze estimation and the mutual gaze detection branches, we achieve better head features than learned by training the mutual gaze detection branch alone. Experimental results on three image datasets show that the proposed approach improves the detection performance significantly without additional annotations. This work also introduces a new image dataset that consists of 33.1K pairs of humans annotated with mutual gaze labels in 29.2K images.

Introduction

Mutual gaze is a cornerstone of human social interaction, signaling everything from intimacy to shared attention. For an AI to understand social scenes, it needs to accurately identify when individuals are making eye contact. However, detecting mutual gaze in "in-the-wild" images is difficult. While datasets labeled with binary "yes/no" mutual gaze status exist, precise 3D gaze vectors—the exact direction the eyes are pointing—are rarely available because collecting such data typically requires specialized eye-tracking hardware in a laboratory setting. This research explores whether the intuitive relationship between 3D gaze and mutual gaze can be leveraged to build a superior detector without requiring expensive 3D ground-truth labels.

Methodology

Multi-Task Learning Framework

The authors propose a multi-task learning framework that utilizes two main branches sharing a single "head image encoder" (a CNN). This shared architecture ensures that the features learned are optimized for both identifying mutual gaze and understanding head orientation.

  1. Mutual Gaze Detection Branch: The primary task, which takes features from two head patches and their spatial relationship to predict a mutual gaze score.
  2. Auxiliary 3D Gaze Estimation Branch: A secondary task that predicts a 3D vector for each head, acting as a regularizer during the training phase.
The proposed multi-task framework featuring a shared head image encoder between the mutual gaze detection and auxiliary 3D gaze estimation branches.

Pseudo-Labels and Geometric Heuristics

Since 3D ground truth is unavailable for in-the-wild datasets, the researchers create "pseudo-labels." For every pair of heads confirmed to be in mutual gaze, the model assumes the 3D gaze vector of Person A must point toward the center of Person B’s head, and vice versa.

To translate 2D image coordinates into 3D space, the following approximations are used:

  • Depth ($Z$): Assumed to be inversely proportional to the square root of the head's bounding box area (where a smaller head indicates a greater distance from the camera).
  • Focal Length ($f$): Approximated based on image dimensions, assuming a standard 53° field of view.
  • 3D Spatial Features: The resulting relative 3D vector is used as an input feature for the detection branch, providing a strong geometric cue that 2D coordinates alone cannot provide.
Geometric derivation of pseudo 3D gaze labels and spatial features from 2D head bounding boxes.

Datasets and Results

The study evaluates the model on three benchmarks:

  • UCO-LAEO: 36k pairs from TV shows.
  • AVA-LAEO: 172k pairs from movie frames.
  • Open-Images-MG (OI-MG): A new dataset introduced by the authors consisting of 33.1K pairs in 29.2K "in-the-wild" images from Flickr, offering more realistic variety than cinematic datasets.

The proposed approach outperformed the previous state-of-the-art (LAEO-Net) across all datasets. On the OI-MG dataset, Average Precision (AP) increased by 10.3 points (from 59.8 to 70.1). Similarly, AP increased by 9.2 points on UCO-LAEO and 2.0 points on AVA-LAEO.

Key Insights

  • Inductive Bias via Geometry: You do not need perfect 3D labels to improve a detector. The coarse geometric assumption (Gaze ≈ Vector to the other head) provides enough of an inductive bias to steer the model toward more meaningful representations of head orientation.
  • Robustness in Low-Data Regimes: The auxiliary task acts as a powerful regularizer. When training data is scarce (e.g., using only 12.5% of the training set), the performance gap between this method and standard approaches widens significantly.

Conclusion

This work demonstrates that incorporating 3D geometric cues through pseudo-labels can significantly boost the performance of image-based mutual gaze detection. While the model relies on coarse depth approximations and can struggle with extreme occlusions or low resolution, the multi-task learning approach provides a scalable way to improve social scene understanding without additional annotation costs. Future research may focus on explicitly processing eye regions or incorporating auxiliary "Visual Focus of Attention" tasks to further refine accuracy.

Summary updated

Abstract

Mutual gaze detection, i.e., predicting whether or not two people are looking at each other, plays an important role in understanding human interactions. In this work, we focus on the task of image-based mutual gaze detection, and propose a simple and effective approach to boost the performance by using an auxiliary 3D gaze estimation task during the training phase. We achieve the performance boost without additional labeling cost by training the 3D gaze estimation branch using pseudo 3D gaze labels deduced from mutual gaze labels. By sharing the head image encoder between the 3D gaze estimation and the mutual gaze detection branches, we achieve better head features than learned by training the mutual gaze detection branch alone. Experimental results on three image datasets show that the proposed approach improves the detection performance significantly without additional annotations. This work also introduces a new image dataset that consists of 33.1K pairs of humans annotated with mutual gaze labels in 29.2K images.
BibTeX
@inproceedings{doosti2021boosting,
  author = {Bardia Doosti and Ching-Hui Chen and Raviteja Vemulapalli and Xuhui Jia and Yukun Zhu and Bradley Green},
  title = {{Boosting Image-based Mutual Gaze Detection using Pseudo 3D Gaze}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=StgiPNrA09W},
}
Figure