Boosting Image-based Mutual Gaze Detection using Pseudo 3D Gaze

Bardia Doosti, Ching-Hui Chen, Raviteja Vemulapalli, Xuhui Jia, Yukun Zhu, Bradley Green

Abstract

Mutual gaze detection, i.e., predicting whether or not two people are looking at each other, plays an important role in understanding human interactions. In this work, we focus on the task of image-based mutual gaze detection, and propose a simple and effective approach to boost the performance by using an auxiliary 3D gaze estimation task during the training phase. We achieve the performance boost without additional labeling cost by training the 3D gaze estimation branch using pseudo 3D gaze labels deduced from mutual gaze labels. By sharing the head image encoder between the 3D gaze estimation and the mutual gaze detection branches, we achieve better head features than learned by training the mutual gaze detection branch alone. Experimental results on three image datasets show that the proposed approach improves the detection performance significantly without additional annotations. This work also introduces a new image dataset that consists of 33.1K pairs of humans annotated with mutual gaze labels in 29.2K images.

Introduction

Mutual gaze is a fundamental building block of human social interaction, signaling intimacy, friendliness, or shared intention. While humans detect this instantly, it remains a difficult computer vision task. The core challenge is that mutual gaze is inherently a 3D phenomenon, but most available datasets only provide 2D images and binary "yes/no" labels. Obtaining precise 3D gaze vectors requires specialized hardware, making it nearly impossible to scale to "in-the-wild" images. Previous methods often relied solely on 2D head positions or complex video temporal cues, which struggle with the subtle geometric nuances of eye contact.

Methodology

The proposed approach utilizes a multi-task learning framework consisting of two main branches that share a common head image encoder.

Overview of the proposed multi-task learning framework for mutual gaze detection.
  1. Mutual Gaze Detection Branch (Primary): This branch takes head image embeddings and spatial features to predict a mutual gaze score. It is used during both training and testing.
  2. 3D Gaze Estimation Branch (Auxiliary): This branch predicts 3D gaze vectors for each head. Crucially, this branch is only used during training to regularize the encoder and help it learn more robust features related to orientation and eye direction.

Pseudo-Label Generation

Since 3D gaze labels are typically unavailable in mutual gaze datasets, the authors introduce a geometric heuristic: if two people are in mutual gaze, their 3D gaze vectors must point toward each other.

Geometric setup for generating pseudo 3D gaze labels from mutual gaze annotations.

The relative 3D position vector between two heads is estimated using a pinhole camera model. Because depth data is missing, depth ($Z$) is approximated as being inversely proportional to the square root of the head's bounding box area ($Z \propto 1/\sqrt{Area}$). For positive mutual gaze samples, this relative 3D vector serves as the "pseudo" ground truth for the 3D gaze branch.

Datasets

The study evaluates the model on three datasets:

  • UCO-LAEO: 36k pairs from TV shows.
  • AVA-LAEO: 172k pairs from the AVA movie dataset.
  • OI-MG (Open-Images Mutual Gaze): A newly introduced dataset containing 33.1K pairs of humans in 29.2K "in-the-wild" images. Unlike video-derived datasets, OI-MG consists of still photos with diverse lighting and poses.

Experimental Results

The proposed approach outperformed the previous state-of-the-art (LAEO-Net) across all benchmarks in terms of Average Precision (AP):

  • OI-MG: 70.1 AP (+10.3 point increase over baseline).
  • UCO-LAEO: 65.1 AP (+9.2 point increase).
  • AVA-LAEO: 72.2 AP (+2.0 point increase).

The auxiliary task proved particularly effective when training data was limited. On the AVA-LAEO dataset, the performance gap between the proposed method and the baseline was significantly wider when using only 12.5% of the training data compared to using 100%, demonstrating the regularizing power of the 3D gaze estimation task.

Conclusion

By incorporating a geometric prior through pseudo 3D gaze labels, this work demonstrates that auxiliary tasks can significantly boost performance in social interaction detection without requiring additional manual annotations. Future work may involve explicit eye-region processing or integrating Visual Focus of Attention (VFA) tasks to further refine gaze detection in challenging conditions.

Summary updated

Abstract

Mutual gaze detection, i.e., predicting whether or not two people are looking at each other, plays an important role in understanding human interactions. In this work, we focus on the task of image-based mutual gaze detection, and propose a simple and effective approach to boost the performance by using an auxiliary 3D gaze estimation task during the training phase. We achieve the performance boost without additional labeling cost by training the 3D gaze estimation branch using pseudo 3D gaze labels deduced from mutual gaze labels. By sharing the head image encoder between the 3D gaze estimation and the mutual gaze detection branches, we achieve better head features than learned by training the mutual gaze detection branch alone. Experimental results on three image datasets show that the proposed approach improves the detection performance significantly without additional annotations. This work also introduces a new image dataset that consists of 33.1K pairs of humans annotated with mutual gaze labels in 29.2K images.
BibTeX
@inproceedings{doosti2021boosting,
  author = {Bardia Doosti and Ching-Hui Chen and Raviteja Vemulapalli and Xuhui Jia and Yukun Zhu and Bradley Green},
  title = {{Boosting Image-based Mutual Gaze Detection using Pseudo 3D Gaze}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=StgiPNrA09W},
}
Figure