When Computer Vision Gazes at Cognition

Tao Gao, Daniel Harari, Joshua B. Tenenbaum, Shimon UllmanarXiv 2014 · arXiv

Introduction

In social environments, the ability to engage in "joint attention"—triangulating exactly what another person is looking at—is a fundamental cognitive skill. While humans are remarkably adept at detecting direct eye contact, our ability to map someone else's gaze onto a specific object in a complex 3D environment is less understood. This paper, a collaborative effort between researchers at MIT and the Weizmann Institute, investigates the performance gap between human observers and computer vision models in a "free-looking" task, where subjects move both their heads and eyes naturally.

The Challenge of Natural Gaze

Most prior research into gaze perception has utilized controlled environments, often fixing the subject's head in place or using simplified 2D stimuli like cartoon characters. However, real-world gaze is a composite of head orientation and eye rotation. This study addresses the challenge of "free" gaze: predicting 3D gaze direction from 2D images when the looker is unconstrained. The core question is whether current computational models can replicate the human ability to integrate these two physical cues.

Methodology

The researchers designed a task involving 52 target objects arranged in concentric arcs on a table. The study utilized a custom-built dataset recorded via a Microsoft Kinect, providing RGB-D data for ground truth measurements of 3D positions and head orientations.

Human Psychophysics

Observers were tasked with identifying which of the 52 targets a "looker" was focusing on. The experiment included two primary conditions: "Eyes-Visible" and "Eyes-Invisible" (where the looker wore sunglasses). This allowed the researchers to isolate the specific contribution of the eyes versus the head pose.

The "Appear-Face-Eyes" Model

To provide a computational baseline, the authors developed an appearance-based model. Unlike traditional models that treat the head and eyes as separate geometric problems, this model uses Histogram of Gradients (HoG) descriptors to represent the face and eye regions. It employs a nearest-neighbor approach, predicting gaze by finding similar facial patches in a training set and calculating a weighted average of their known 3D orientations. This was compared against baselines that ignored eye data or relied solely on linear regression from raw head-tracking data.

Key Results and Comparative Analysis

Human Accuracy and the "12° Rule"

In the free-looking task, humans achieved an exact target accuracy of approximately 40%. However, when the margin of error was expanded to include adjacent targets (one row or column off), accuracy jumped to 80-90%. Most human errors fell within a 12° visual angle. Interestingly, humans were significantly better at left-right (column) discrimination when the looker faced them, though this advantage did not extend to up-down (row) discrimination.

The Human-Machine Gap

Humans significantly outperformed all computational models. Even the specialized "Appear-Face-Eyes" model struggled to match human performance in the Eyes-Visible condition. A critical finding was the "15% Eye Boost": human accuracy dropped by 15% when the looker's eyes were hidden. This confirms that while head pose provides a general orientation, the eyes are essential for high-resolution joint attention.

Bias vs. Variance

When analyzing model failures, the researchers discovered that the models' variance (consistency) was actually comparable to or better than that of humans. The models were not "noisy"; rather, they were consistently incorrect. Their failure stemmed from high systematic bias—a fundamental difficulty in mapping specific facial appearances to correct 3D coordinates across different individuals.

The Legibility Paradox

One of the study's most intriguing insights was the high variance in "gaze legibility" between different lookers. Certain individuals were significantly easier for both humans and models to read. This suggests that legibility may be an intrinsic physical trait determined by face shape or specific eye-movement patterns. The computational model successfully captured this variance, suggesting that the difficulty of joint attention is rooted in the physical geometry of the face.

Conclusion and Future Directions

The study highlights a significant gap in how machines interpret the nuances of human gaze. While the authors' model avoided "rectifying" eye images (warping them to a frontal view) to maintain cognitive plausibility, they acknowledge that better representations of the eye region are needed to handle "inter-pose" variations—how the appearance of an eye changes relative to head tilt.

Ultimately, the mystery of why some people are more "legible" than others remains an open question. Bridging this gap will require not only better algorithms but a deeper understanding of the social psychology and physical geometry that govern how we signal attention to one another.

Summary updated

Abstract

Joint attention is a core, early-developing form of social interaction. It is based on our ability to discriminate the third party objects that other people are looking at. While it has been shown that people can accurately determine whether another person is looking directly at them versus away, little is known about human ability to discriminate a third person gaze directed towards objects that are further away, especially in unconstraint cases where the looker can move her head and eyes freely. In this paper we address this question by jointly exploring human psychophysics and a cognitively motivated computer vision model, which can detect the 3D direction of gaze from 2D face images. The synthesis of behavioral study and computer vision yields several interesting discoveries. (1) Human accuracy of discriminating targets 8{\deg}-10{\deg} of visual angle apart is around 40% in a free looking gaze task; (2) The ability to interpret gaze of different lookers vary dramatically; (3) This variance can be captured by the computational model; (4) Human outperforms the current model significantly. These results collectively show that the acuity of human joint attention is indeed highly impressive, given the computational challenge of the natural looking task. Moreover, the gap between human and model performance, as well as the variability of gaze interpretation across different lookers, require further understanding of the underlying mechanisms utilized by humans for this challenging task.
BibTeX
@misc{gao2014when,
  author = {Tao Gao and Daniel Harari and Joshua B. Tenenbaum and Shimon Ullman},
  title = {{When Computer Vision Gazes at Cognition}},
  year = {2014},
  eprint = {1412.2672},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1412.2672},
}
Figure