Measuring and modeling the perception of natural and unconstrained gaze in humans and machines

Daniel Harari, Tao Gao, Nancy Kanwisher, Joshua B. Tenenbaum, Shimon UllmanarXiv 2016 · arXiv

Abstract

Gaze perception is a fundamental social signal, yet its mechanisms in unconstrained, naturalistic settings remain poorly understood. This study investigates how humans and machines estimate gaze direction when both head and eye movements are involved. We present a two-stage computational model that first estimates head pose and then gaze direction, significantly outperforming end-to-end deep learning approaches. Our findings highlight the critical role of the eye region and suggest that gaze perception relies on a hierarchical coordinate system rather than simple pattern matching.

Introduction

Gaze serves as a "silent language" in social interaction, conveying interest, intent, and attention. While traditional research often uses constrained setups (e.g., fixed head positions), real-world gaze is "unconstrained," involving coordinated head and eye movements. This creates a significant computational challenge: the same eye position relative to the socket can indicate different gaze directions depending on head orientation. This research, conducted by teams from MIT and the Weizmann Institute, explores how humans maintain accuracy in these scenarios and proposes a machine learning architecture that mimics this capability.

Methodology

The study combined human psychophysics with computational modeling to dissect the mechanics of gaze perception.

Human Experiments

Researchers established a "looker" and "observer" paradigm. The looker focused on specific targets among an array of 52 objects arranged in a concentric layout. Observers were tasked with identifying the target under several conditions:

  • Live: Face-to-face interaction.
  • Recorded: Video playback of the looker.
  • Degraded: Conditions where specific facial features (e.g., the nose bridge or eyes) were isolated or blurred.

Computational Modeling

The team developed a two-stage model to test the hypothesis that gaze perception is hierarchical:

  1. Stage 1: Estimate the 3D head pose from the face image.
  2. Stage 2: Estimate gaze direction from the eyes, conditioned on the head pose determined in the first stage.

This approach was compared against state-of-the-art end-to-end Deep Neural Networks (DNNs) that attempt to predict gaze directly from a full image in a single step.

Dataset and Evaluation

A custom dataset was generated using a Microsoft Kinect V2 RGB-D sensor. It included 1,916 face images from 10 participants, each annotated with precise 3D head orientation and target coordinates. Models were evaluated using "leave-one-looker-out" cross-validation to ensure generalizability to novel faces.

Results

The study yielded several key insights into the nature of gaze perception:

  • The Primacy of the Eye Region: Both humans and the two-stage model performed nearly as well when viewing only a narrow strip around the eyes as they did with the full face. This "eyes-region" contains sufficient geometric information to solve for both head pose and eye direction.
  • Hierarchical Processing: The two-stage k-Nearest Neighbor (k-NN) model achieved an accuracy of 19.15%, significantly outperforming the end-to-end DNN, which reached only 8.25%. This suggests that gaze perception involves establishing a coordinate system (the head) before interpreting the ocular signal.
  • Human Performance Gap: In "Live" conditions, humans achieved a 46% exact target hit rate, which dropped to 31% in "Recorded" conditions. The researchers concluded that this discrepancy is likely due to 3D depth cues (stereopsis) available in person, rather than the dynamics of motion.
  • Angular Error: The two-stage model's mean angular error was approximately 8.4°, nearing human performance levels in recorded settings.

Discussion

The model successfully replicated the Wollaston illusion, a classic perceptual effect where the same eye image appears to look in different directions based on the surrounding head orientation. Notably, this replication only occurred when the model was specifically trained for gaze estimation, suggesting the human brain possesses specialized, gaze-selective neural machinery.

Limitations and Future Directions

While the two-stage model represents a significant advancement, it currently operates on 2D images and cannot yet match the superior performance humans exhibit in "Live" 3D settings. Future research should focus on:

  1. Depth Integration: Incorporating stereoscopic or depth data into computational models.
  2. Dataset Diversity: Expanding the training data to include a wider variety of ethnicities, ages, and lighting conditions.
  3. Social Complexity: Moving beyond static targets to model "triadic" interactions and joint attention in dynamic social environments.

Summary updated

Abstract

Humans are remarkably adept at interpreting the gaze direction of other individuals in their surroundings. This skill is at the core of the ability to engage in joint visual attention, which is essential for establishing social interactions. How accurate are humans in determining the gaze direction of others in lifelike scenes, when they can move their heads and eyes freely, and what are the sources of information for the underlying perceptual processes? These questions pose a challenge from both empirical and computational perspectives, due to the complexity of the visual input in real-life situations. Here we measure empirically human accuracy in perceiving the gaze direction of others in lifelike scenes, and study computationally the sources of information and representations underlying this cognitive capacity. We show that humans perform better in face-to-face conditions compared with recorded conditions, and that this advantage is not due to the availability of input dynamics. We further show that humans are still performing well when only the eyes-region is visible, rather than the whole face. We develop a computational model, which replicates the pattern of human performance, including the finding that the eyes-region contains on its own, the required information for estimating both head orientation and direction of gaze. Consistent with neurophysiological findings on task-specific face regions in the brain, the learned computational representations reproduce perceptual effects such as the Wollaston illusion, when trained to estimate direction of gaze, but not when trained to recognize objects or faces.
BibTeX
@misc{harari2016measuring,
  author = {Daniel Harari and Tao Gao and Nancy Kanwisher and Joshua B. Tenenbaum and Shimon Ullman},
  title = {{Measuring and modeling the perception of natural and unconstrained gaze in humans and machines}},
  year = {2016},
  eprint = {1611.09819},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1611.09819},
}
Figure