Gaze Perception in Humans and CNN-Based Model

Nicole X. Han, William Yang Wang, Miguel P. EcksteinarXiv 2021 · OpenReview

TL;DR

This paper investigates how well Convolutional Neural Networks (CNNs) can infer human gaze direction compared to actual humans, finding that while CNNs are improving, they still lag behind humans in leveraging contextual scene information. The study highlights that humans are more adept at using cues like the presence of an attended target and the number of gazers to refine their gaze estimations, suggesting future AI models need to incorporate richer contextual understanding.

What Problem Does This Paper Address?

Accurately predicting where someone is looking (gaze perception) is fundamental for human social interaction and crucial for developing AI that can interact naturally with us. While AI, particularly CNNs, has made strides in visual tasks, it's unclear how well these models truly understand and replicate human gaze perception, especially in complex, real-world scenarios involving multiple people. Existing gaze estimation models often focus on individual faces or limited visual cues, potentially missing the nuanced contextual understanding that humans effortlessly employ. This paper aims to bridge this gap by directly comparing human and CNN-based gaze estimation performance on realistic images and identifying key differences in their decision-making processes.

Methods

The researchers created a dataset of image frames from videos where multiple individuals were instructed to look at a designated target. To create variations, they manipulated the presence of the target and the number of "gaze-orienting" individuals (those looking at the target). For evaluation, they used a state-of-the-art CNN model pre-trained for predicting attended targets. This model takes an image and a cropped head region as input and outputs a probability map of the attended location. Human participants estimated gaze by clicking on images, and their responses were aggregated. The study then compared the accuracy of human and model estimations, analyzed error correlations, and investigated the influence of scene context (number of gazers, target presence) on their performance.

Datasets

The study utilized a custom dataset of image frames extracted from videos originally recorded at the University of California, Santa Barbara. These videos depicted various indoor and outdoor scenes with multiple individuals looking at a common target. The dataset was specifically designed to allow for manipulation of two key contextual factors:

  • Number of Gaze-Orienting Individuals: Images were created with one, two, or three individuals looking at the target.
  • Presence of the Attended Target: Images were generated with and without the designated target person visible.

Key Insights and Ideas

  • Context is King for Humans: Humans significantly leverage scene context, such as the presence of the target and the number of people looking, to improve their gaze estimations. CNNs, while capable, are less sensitive to these contextual cues.
  • Human Gaze is More Consistent: Humans exhibit more consistent gaze estimations across different images compared to the CNN model, suggesting a more robust and generalized understanding.
  • Target Presence Boosts Performance for Both, But More for Humans: The presence of the attended target significantly improves gaze estimation accuracy for both humans and the model. However, this benefit is much more pronounced for humans, indicating they use this cue more effectively.
  • Multiple Gazers Help Humans, Not the Model: Humans can integrate information from multiple gaze-orienting individuals to refine their estimations, whereas the CNN model does not show a significant improvement when more people are looking.

Key Results

  • Overall Accuracy: While not explicitly quantified with a single metric, the paper implies humans generally perform well, and the model's performance is comparable but with notable differences in how context influences it.
  • Correlation: The correlation between human-human gaze estimations (r = 0.55) was comparable to human-model estimations (r = 0.50), suggesting the model captures some aspects of human gaze perception.
  • Effect of Number of Gazers: Human estimation error increased significantly when only one individual was looking compared to two or three (p < 0.001). The model's error was not significantly affected by the number of gazers.
  • Effect of Target Presence: Both humans and the model had significantly lower errors when the target was present (p < 0.001). Humans showed a much larger benefit from the target's presence compared to the model.
  • Error Direction: Humans showed significantly higher vertical error than the model, but when the target was present, humans' horizontal errors were significantly smaller than the model's, highlighting the model's lack of sensitivity to the target's influence.

Limitations and Future Work

  • Model Limitations: The current CNN model processes only one gaze-orienting person's head region at a time, which is a significant limitation preventing it from effectively integrating information from multiple individuals. The paper suggests that future models need to be able to integrate more context and multiple cues.
  • Future Directions: The authors advocate for developing models that can better integrate contextual information and multiple sources of cues to achieve performance closer to human capabilities in gaze estimation. This includes exploring how models can better utilize the presence of attended targets and the social dynamics of multiple gazers.

Summary updated

Abstract

Making accurate inferences about other individuals' locus of attention is essential for human social interactions and will be important for AI to effectively interact with humans. In this study, we compare how a CNN (convolutional neural network) based model of gaze and humans infer the locus of attention in images of real-world scenes with a number of individuals looking at a common location. We show that compared to the model, humans' estimates of the locus of attention are more influenced by the context of the scene, such as the presence of the attended target and the number of individuals in the image.
BibTeX
@misc{han2021gaze,
  author = {Nicole X. Han and William Yang Wang and Miguel P. Eckstein},
  title = {{Gaze Perception in Humans and CNN-Based Model}},
  year = {2021},
  eprint = {2104.08447},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.08447},
}

Loading related content...

Figure