A Synchronized Multi-Modal Attention-Caption Dataset and Analysis

Sen He, Hamed R. Tavakoli, Ali Borji, Nicolas PugeaultarXiv 2019 · OpenReview

TL;DR

This paper introduces a novel dataset of synchronized eye-tracking and image descriptions to investigate how humans attend to images while generating captions. They find that human attention differs significantly between free-viewing and captioning tasks, and crucially, that current "soft attention" mechanisms in AI captioning models don't fully replicate human attention patterns, though integrating visual saliency can improve performance.

What Problem Does This Paper Address?

Automatic image captioning, where AI generates textual descriptions for images, has seen impressive progress. Many modern captioning models employ "soft attention" mechanisms, which are often presented as mimicking human visual attention. However, there's a fundamental question: do these AI mechanisms truly reflect how humans look at and describe images? This paper aims to bridge this gap by directly comparing human attention patterns (measured by eye-tracking) with AI attention during the image captioning task. They also explore how human attention relates to the language used in descriptions and whether incorporating visual saliency, a bottom-up attention cue, can enhance AI captioning performance.

Methods

The core of this work is the creation of a new dataset and subsequent analyses.

  • Dataset Creation: The authors collected synchronized data of participants viewing images, describing them verbally, and having their eye movements recorded. This allows for a direct comparison of what humans look at and what they say about an image simultaneously.
  • Human Attention Analysis: They compared attention patterns in two conditions: free-viewing (just looking at an image) and image captioning (describing the image). They analyzed differences in fixation locations, dispersion, and inter-observer consistency.
  • Attention-Description Relationship: The study investigated how fixations on objects relate to whether those objects are mentioned in the generated descriptions. They also looked at the order in which objects are described and their corresponding attention.
  • Human vs. Machine Attention: The paper compares human attention maps with the internal feature activations of a CNN used as an image encoder in captioning models. They also analyze the spatial and temporal consistency between human attention and the soft-attention mechanisms in AI captioning models.
  • Saliency Integration: A novel architecture is proposed that combines a saliency prediction module (SPM) with a standard soft-attention captioning model. This SPM guides the attention mechanism to focus on salient regions identified by bottom-up cues.

Datasets

  • Newly Introduced Dataset: The paper introduces a novel dataset consisting of synchronized eye-movements and verbal descriptions recorded over images. This dataset comprises two corpora:
  • capgazel: 1,000 images from the Pascal-50S dataset, with 5 subjects providing descriptions and eye-tracking data.
  • capgaze2: 3,000 images from MSCOCO, with 11 subjects providing data (each image recorded from 3 subjects).
  • Existing Datasets Used:
    • Pascal-50S: Used as the source for images in the capgazel corpus.
  • MSCOCO: Used as the source for images in the capgaze2 corpus.
  • Flickr30k & MSCOCO: Used for training and evaluating the proposed captioning model.
  • Salicon: Used for training the saliency prediction model in the proposed architecture.
  • sbugaze: Used for comparison of human attention in free-viewing vs. captioning tasks.

Key Insights and Ideas

  • Human Attention is Task-Dependent: Humans don't look at images the same way when they're just browsing versus when they need to describe them. During captioning, attention is more dispersed, covering a wider variety of regions.
  • Strong Link Between Gaze and Words: What humans describe is highly correlated with where they look. A remarkable 97% of described objects are attended to, suggesting a direct mapping between visual focus and linguistic output.
  • AI Attention Isn't Quite Human: Current soft-attention mechanisms in image captioning models, while effective, don't perfectly mirror human attention. They differ both spatially (where the attention is focused) and temporally (when it's focused there). This suggests a significant gap in how machines "understand" visual content for description.
  • Saliency Matters: Bottom-up visual saliency (what naturally grabs attention) can be a powerful cue. Integrating this with top-down attention mechanisms in AI captioning models leads to significant performance improvements.

Key Results

  • Attention Differences: Human attention in captioning tasks is more dispersed than in free-viewing. Inter-observer congruency (how much different people look at the same things) is lower in captioning than in free-viewing.
  • Object-Description Correlation: 97% of described objects are attended to. However, many attended objects are not described, indicating that attention is broader than just what's explicitly mentioned.
  • CNN Encoder Alignment: CNN feature encoders align reasonably well with human-attended regions (around 78%), suggesting they capture some important visual cues.
  • Human vs. Machine Attention Discrepancy: Spatial and temporal consistency between human and machine attention in captioning is low. Crucially, the similarity of machine attention to human attention did not correlate with the quality of the generated captions (CIDEr score).
  • Saliency Boost: Integrating a saliency prediction module into a soft-attention captioning model significantly improved performance on Flickr30k (up to 18.2% improvement in CIDEr) and MSCOCO (up to 12% improvement in CIDEr) compared to a baseline soft-attention model.

Limitations and Future Work

  • Dataset Size: While novel and synchronized, the dataset size, particularly for the capgazel corpus (1000 images), might be considered moderate for training very large models from scratch.
  • Human-AI Attention Gap: The paper highlights a significant gap between human and machine attention patterns. Future work could focus on developing AI attention mechanisms that more closely mimic human cognitive processes.
  • Saliency Integration: The proposed architecture shows promise by integrating saliency. Further research could explore more sophisticated ways to combine bottom-up and top-down attention, potentially leading to even better captioning performance and more human-like visual understanding.
  • Generalizability: The authors suggest their architecture could be applied to other tasks where visual saliency is important, opening avenues for future research beyond image captioning.

Summary updated

Abstract

In this work, we present a novel dataset consisting of eye movements and verbal descriptions recorded synchronously over images. Using this data, we study the differences in human attention during free-viewing and image captioning tasks. We look into the relationship between human attention and language constructs during perception and sentence articulation. We also analyse attention deployment mechanisms in the top-down soft attention approach that is argued to mimic human attention in captioning tasks, and investigate whether visual saliency can help image captioning. Our study reveals that (1) human attention behaviour differs in free-viewing and image description tasks. Humans tend to fixate on a greater variety of regions under the latter task, (2) there is a strong relationship between described objects and attended objects ($97\%$ of the described objects are being attended), (3) a convolutional neural network as feature encoder accounts for human-attended regions during image captioning to a great extent (around $78\%$), (4) soft-attention mechanism differs from human attention, both spatially and temporally, and there is low correlation between caption scores and attention consistency scores. These indicate a large gap between humans and machines in regards to top-down attention, and (5) by integrating the soft attention model with image saliency, we can significantly improve the model's performance on Flickr30k and MSCOCO benchmarks. The dataset can be found at: https://github.com/SenHe/Human-Attention-in-Image-Captioning.
BibTeX
@misc{he2019synchronized,
  author = {Sen He and Hamed R. Tavakoli and Ali Borji and Nicolas Pugeault},
  title = {{A Synchronized Multi-Modal Attention-Caption Dataset and Analysis}},
  year = {2019},
  eprint = {1903.02499},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1903.02499},
}
Figure