TL;DR
This paper introduces a novel dataset of synchronized eye-tracking and image descriptions to investigate how humans attend to images while generating captions. They find that human attention differs significantly between free-viewing and captioning tasks, and crucially, that current "soft attention" mechanisms in AI captioning models don't fully replicate human attention patterns, though integrating visual saliency can improve performance.
What Problem Does This Paper Address?
Automatic image captioning, where AI generates textual descriptions for images, has seen impressive progress. Many modern captioning models employ "soft attention" mechanisms, which are often presented as mimicking human visual attention. However, there's a fundamental question: do these AI mechanisms truly reflect how humans look at and describe images? This paper aims to bridge this gap by directly comparing human attention patterns (measured by eye-tracking) with AI attention during the image captioning task. They also explore how human attention relates to the language used in descriptions and whether incorporating visual saliency, a bottom-up attention cue, can enhance AI captioning performance.
Methods
The core of this work is the creation of a new dataset and subsequent analyses.
- Dataset Creation: The authors collected synchronized data of participants viewing images, describing them verbally, and having their eye movements recorded. This allows for a direct comparison of what humans look at and what they say about an image simultaneously.
- Human Attention Analysis: They compared attention patterns in two conditions: free-viewing (just looking at an image) and image captioning (describing the image). They analyzed differences in fixation locations, dispersion, and inter-observer consistency.
- Attention-Description Relationship: The study investigated how fixations on objects relate to whether those objects are mentioned in the generated descriptions. They also looked at the order in which objects are described and their corresponding attention.
- Human vs. Machine Attention: The paper compares human attention maps with the internal feature activations of a CNN used as an image encoder in captioning models. They also analyze the spatial and temporal consistency between human attention and the soft-attention mechanisms in AI captioning models.
- Saliency Integration: A novel architecture is proposed that combines a saliency prediction module (SPM) with a standard soft-attention captioning model. This SPM guides the attention mechanism to focus on salient regions identified by bottom-up cues.
Datasets
- Newly Introduced Dataset: The paper introduces a novel dataset consisting of synchronized eye-movements and verbal descriptions recorded over images. This dataset comprises two corpora:
capgazel: 1,000 images from the Pascal-50S dataset, with 5 subjects providing descriptions and eye-tracking data.capgaze2: 3,000 images from MSCOCO, with 11 subjects providing data (each image recorded from 3 subjects).- Existing Datasets Used:
- Pascal-50S: Used as the source for images in the
capgazelcorpus.
- Pascal-50S: Used as the source for images in the
- MSCOCO: Used as the source for images in the
capgaze2corpus. - Flickr30k & MSCOCO: Used for training and evaluating the proposed captioning model.
- Salicon: Used for training the saliency prediction model in the proposed architecture.
- sbugaze: Used for comparison of human attention in free-viewing vs. captioning tasks.
Key Insights and Ideas
- Human Attention is Task-Dependent: Humans don't look at images the same way when they're just browsing versus when they need to describe them. During captioning, attention is more dispersed, covering a wider variety of regions.
- Strong Link Between Gaze and Words: What humans describe is highly correlated with where they look. A remarkable 97% of described objects are attended to, suggesting a direct mapping between visual focus and linguistic output.
- AI Attention Isn't Quite Human: Current soft-attention mechanisms in image captioning models, while effective, don't perfectly mirror human attention. They differ both spatially (where the attention is focused) and temporally (when it's focused there). This suggests a significant gap in how machines "understand" visual content for description.
- Saliency Matters: Bottom-up visual saliency (what naturally grabs attention) can be a powerful cue. Integrating this with top-down attention mechanisms in AI captioning models leads to significant performance improvements.
Key Results
- Attention Differences: Human attention in captioning tasks is more dispersed than in free-viewing. Inter-observer congruency (how much different people look at the same things) is lower in captioning than in free-viewing.
- Object-Description Correlation: 97% of described objects are attended to. However, many attended objects are not described, indicating that attention is broader than just what's explicitly mentioned.
- CNN Encoder Alignment: CNN feature encoders align reasonably well with human-attended regions (around 78%), suggesting they capture some important visual cues.
- Human vs. Machine Attention Discrepancy: Spatial and temporal consistency between human and machine attention in captioning is low. Crucially, the similarity of machine attention to human attention did not correlate with the quality of the generated captions (CIDEr score).
- Saliency Boost: Integrating a saliency prediction module into a soft-attention captioning model significantly improved performance on Flickr30k (up to 18.2% improvement in CIDEr) and MSCOCO (up to 12% improvement in CIDEr) compared to a baseline soft-attention model.
Limitations and Future Work
- Dataset Size: While novel and synchronized, the dataset size, particularly for the
capgazelcorpus (1000 images), might be considered moderate for training very large models from scratch. - Human-AI Attention Gap: The paper highlights a significant gap between human and machine attention patterns. Future work could focus on developing AI attention mechanisms that more closely mimic human cognitive processes.
- Saliency Integration: The proposed architecture shows promise by integrating saliency. Further research could explore more sophisticated ways to combine bottom-up and top-down attention, potentially leading to even better captioning performance and more human-like visual understanding.
- Generalizability: The authors suggest their architecture could be applied to other tasks where visual saliency is important, opening avenues for future research beyond image captioning.
Summary updated