Human Attention in Image Captioning: Dataset and Analysis

Sen He, Hamed Rezazadegan Tavakoli, Ali Borji, Nicolas PugeaultICCV 2019 · OpenReview

Abstract

This work introduces a novel dataset of synchronous eye movements and verbal descriptions recorded during image viewing. By analyzing this data, researchers explored the gap between human attention and the "soft-attention" mechanisms commonly used in AI image captioning models. The study reveals significant differences in attention patterns between free-viewing and captioning tasks and demonstrates that integrating human-like visual saliency can substantially improve AI performance.

Introduction: The Soft-Attention Assumption

In computer vision, attention mechanisms are designed to help models focus on relevant image regions when generating specific words. While often marketed as "mimicking human attention," this claim has lacked empirical validation. Previous research either used small samples or recorded gaze and language asynchronously. This paper addresses this gap by providing a large-scale dataset of simultaneous gaze and speech to evaluate how well AI models actually replicate human visual strategies.

The Human-Attention-in-Image-Captioning Dataset

The researchers developed a specialized setup using a Tobii X2-30 eye-tracker and voice recording to capture real-time data. The resulting dataset includes 4,000 images (from Pascal-50S and MSCOCO) with 14,000 instances of synchronous eye-tracking and verbal descriptions.

Example of synchronous recording: image, audio, text, and eye-fixation sequence.

This dataset allows for a granular look at how human gaze moves in direct correlation with the words being spoken, providing a ground truth for temporal and spatial attention in captioning.

Key Insights: Task-Driven Attention

A primary finding is that human attention is highly task-dependent. When subjects are asked to describe an image (captioning), their gaze is more dispersed compared to simple free-viewing. To generate a coherent sentence, humans must scan a wider variety of regions to establish context and relationships between objects.

Comparison of attention maps for free-viewing (a) versus captioning tasks (b, c).

Other critical observations include:

  • Object Correlation: 97% of described objects are attended to by the viewer.
  • CNN Alignment: Standard pre-trained CNNs (like VGG-16) naturally capture about 78% of human-attended regions, suggesting they are effective feature encoders for this task.
  • The Soft-Attention Gap: Current AI "soft-attention" models do not align well with human gaze spatially or temporally. Surprisingly, there is little correlation between how "human-like" a model's attention is and its final captioning score.

Improving AI with Visual Saliency

Because top-down soft-attention (predicting what to look at based on the next word) proved inconsistent with human behavior, the authors proposed a bottom-up saliency-guided approach. They integrated a Saliency Prediction Module (SPM) to identify naturally eye-catching regions.

This approach yielded significant performance gains over standard baselines:

  • Flickr30k: 18.2% improvement in CIDEr score.
  • MSCOCO: 12% improvement in CIDEr score.

Qualitatively, the saliency-guided models were better at filtering out background noise and focusing on primary subjects, much like a human observer.

Conclusion and Future Work

The study highlights a significant gap between human visual perception and current AI attention mechanisms. While AI can be improved by incorporating human-like saliency, the temporal sequence of AI attention (the "scanpath") still differs from humans. Future research should explore more sophisticated ways to integrate human-like visual search strategies into the captioning pipeline to bridge this divide.

Summary updated

Abstract

In this work, we present a novel dataset consisting of eye movements and verbal descriptions recorded synchronously over images. Using this data, we study the differences in human attention during free-viewing and image captioning tasks. We look into the relationship between human atten- tion and language constructs during perception and sen- tence articulation. We also analyse attention deployment mechanisms in the top-down soft attention approach that is argued to mimic human attention in captioning tasks, and investigate whether visual saliency can help image caption- ing. Our study reveals that (1) human attention behaviour differs in free-viewing and image description tasks. Hu- mans tend to fixate on a greater variety of regions under the latter task, (2) there is a strong relationship between de- scribed objects and attended objects (97% of the described objects are being attended), (3) a convolutional neural net- work as feature encoder accounts for human-attended re- gions during image captioning to a great extent (around 78%), (4) soft-attention mechanism differs from human at- tention, both spatially and temporally, and there is low correlation between caption scores and attention consis- tency scores. These indicate a large gap between humans and machines in regards to top-down attention, and (5) by integrating the soft attention model with image saliency, we can significantly improve the model's performance on Flickr30k and MSCOCO benchmarks. The dataset can be found at: https://github.com/SenHe/ Human-Attention-in-Image-Captioning.
BibTeX
@inproceedings{he2019human,
  author = {Sen He and Hamed Rezazadegan Tavakoli and Ali Borji and Nicolas Pugeault},
  title = {{Human Attention in Image Captioning: Dataset and Analysis}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=Ujj5a0vQDf},
}
Figure