An Evaluation of Image-Based Verb Prediction Models against Human Eye-Tracking Data

Spandana Gella, Frank KellerNAACL 2018 · DOI

Introduction

A central question in artificial intelligence is whether deep neural networks perceive the visual world in a manner analogous to human vision. While modern computer vision systems have achieved remarkable success in object recognition (identifying nouns), the task of action recognition (identifying verbs) remains significantly more complex. Actions are often distributed across an image and require an understanding of the relationship between objects and their environment.

Researchers frequently express concern that these models function as "black boxes," potentially relying on irrelevant statistical shortcuts rather than genuine visual cues. A prominent study by Das et al. (2016) suggested that Visual Question Answering (VQA) models do not align with human visual attention. This paper investigates whether this discrepancy persists in the specific domain of verb prediction: do the regions a model identifies as salient for a verb correlate with where a human looks when performing the same task?

Methodology

The study evaluates a verb prediction model utilizing a ResNet-152 architecture, trained as a multilabel CNN to predict the 250 most common verbs from datasets such as TUHOI, Flickr30k, and COCO. To interpret the model's internal focus, the authors employed Class Activation Mapping (CAM), which projects final-layer weights back onto convolutional feature maps to generate heatmaps of influential regions.

The model (M) was compared against two primary baselines:

  1. Center Bias (CB): A simple heuristic assuming attention is always directed at the center of the frame.
  2. Visual Salience (SM): A model (DSCLRCN) designed to predict bottom-up attention based on low-level features like color and contrast.

Evaluation was conducted using Spearman’s rank correlation against the PASCAL VOC 2012 Actions Fixation dataset, which contains ground-truth eye-tracking data from humans tasked with identifying specific actions.

Qualitative comparison of attention heatmaps for human observers (H), center bias (CB), visual saliency (SM), and the verb prediction model (M) across various action classes.

Results and Discussion

The experimental results demonstrate a significant alignment between the verb prediction model and human attention. While human observers showed a high internal consistency (correlation of 0.923), the verb model (M) achieved a correlation of 0.529, substantially outperforming the general visual salience baseline (0.344).

Rank correlation coefficients across different action verbs, comparing the model and baselines against human eye-tracking data.

A key finding is the effectiveness of combining the model with center bias (CB+M), which yielded the highest correlation of 0.628. This suggests that the model captures task-specific visual cues that complement the natural human tendency to focus on the center of an image.

Furthermore, the study highlights that attention is goal-directed. The model's heatmaps shifted based on the target verb even when viewing the same image, mirroring the "top-down" nature of human perception. This contrasts with the findings of Das et al. (2016), suggesting that the task of labeling an action forces a model to learn visual features more closely aligned with human intuition than the broader task of answering open-ended questions.

Conclusion

This research provides evidence that neural networks trained for action recognition are not merely exploiting statistical shortcuts but are developing visual strategies that mirror human perception. While general saliency is a poor predictor of task-based attention, the verb prediction model successfully identifies relevant regions for action classification. Future work may explore the causal necessity of these regions by masking them to observe performance degradation, further clarifying the link between attention and accuracy.

Summary updated

BibTeX
@inproceedings{gella2018evaluation,
  author = {Spandana Gella and Frank Keller},
  title = {{An Evaluation of Image-Based Verb Prediction Models against Human Eye-Tracking Data}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.18653/v1/n18-2119},
}
Figure