Abstract
This work introduces a novel dataset of synchronous eye movements and verbal descriptions recorded during image viewing. By analyzing this data, researchers explored the gap between human attention and the "soft-attention" mechanisms commonly used in AI image captioning models. The study reveals significant differences in attention patterns between free-viewing and captioning tasks and demonstrates that integrating human-like visual saliency can substantially improve AI performance.
Introduction: The Soft-Attention Assumption
In computer vision, attention mechanisms are designed to help models focus on relevant image regions when generating specific words. While often marketed as "mimicking human attention," this claim has lacked empirical validation. Previous research either used small samples or recorded gaze and language asynchronously. This paper addresses this gap by providing a large-scale dataset of simultaneous gaze and speech to evaluate how well AI models actually replicate human visual strategies.
The Human-Attention-in-Image-Captioning Dataset
The researchers developed a specialized setup using a Tobii X2-30 eye-tracker and voice recording to capture real-time data. The resulting dataset includes 4,000 images (from Pascal-50S and MSCOCO) with 14,000 instances of synchronous eye-tracking and verbal descriptions.
This dataset allows for a granular look at how human gaze moves in direct correlation with the words being spoken, providing a ground truth for temporal and spatial attention in captioning.
Key Insights: Task-Driven Attention
A primary finding is that human attention is highly task-dependent. When subjects are asked to describe an image (captioning), their gaze is more dispersed compared to simple free-viewing. To generate a coherent sentence, humans must scan a wider variety of regions to establish context and relationships between objects.
Other critical observations include:
- Object Correlation: 97% of described objects are attended to by the viewer.
- CNN Alignment: Standard pre-trained CNNs (like VGG-16) naturally capture about 78% of human-attended regions, suggesting they are effective feature encoders for this task.
- The Soft-Attention Gap: Current AI "soft-attention" models do not align well with human gaze spatially or temporally. Surprisingly, there is little correlation between how "human-like" a model's attention is and its final captioning score.
Improving AI with Visual Saliency
Because top-down soft-attention (predicting what to look at based on the next word) proved inconsistent with human behavior, the authors proposed a bottom-up saliency-guided approach. They integrated a Saliency Prediction Module (SPM) to identify naturally eye-catching regions.
This approach yielded significant performance gains over standard baselines:
- Flickr30k: 18.2% improvement in CIDEr score.
- MSCOCO: 12% improvement in CIDEr score.
Qualitatively, the saliency-guided models were better at filtering out background noise and focusing on primary subjects, much like a human observer.
Conclusion and Future Work
The study highlights a significant gap between human visual perception and current AI attention mechanisms. While AI can be improved by incorporating human-like saliency, the temporal sequence of AI attention (the "scanpath") still differs from humans. Future research should explore more sophisticated ways to integrate human-like visual search strategies into the captioning pipeline to bridge this divide.
Summary updated