TL;DR
This paper bridges the gap between how AI models "see" videos and how humans do, by comparing AI attention maps to human eye-tracking data. They found that a simple CNN+Transformer model, trained solely to predict video memorability, surprisingly mimics human attention patterns, focusing on similar objects and temporal dynamics, especially for memorable videos.
What Problem Does This Paper Address?
Predicting what makes a video memorable is crucial for applications in education and advertising. While previous work has focused on building models to predict memorability scores, there's been a lack of deep understanding of why these models work and how their internal "attention" mechanisms align with human perception. Specifically, do AI models learn to focus on the same visual elements and temporal cues that humans do when trying to remember a video? This paper aims to answer this by directly comparing AI attention with human gaze data.
Methods
The authors propose a CNN+Transformer architecture for video memorability prediction.
- Image Encoder: They use a pre-trained CLIP model as a backbone to extract spatio-temporal features from video frames.
- Transformer Encoder: These features are then fed into a Transformer encoder, which allows for self-attention across space and time. A special "CLS" token is used to aggregate information for the final prediction.
- Memorability Prediction: A simple MLP head takes the CLS token's representation to predict the memorability score.
- Attention Extraction: Crucially, they extract the self-attention weights from the Transformer's last layer to generate attention maps.
To compare with human perception, they conducted an eye-tracking study. Participants watched videos and performed a memory task, while their gaze was recorded. This data was processed into fixation density maps.
Datasets
- Memento10K: A dataset of 10K video clips (3 seconds each) with human annotations for memorability.
- VideoMem: A dataset of 10K video clips (7 seconds each) with human annotations for memorability.
The authors also mention using the LaMem dataset for image memorability pretraining and FIGRIM for image dataset fine-tuning.
Key Insights and Ideas
- AI Mimics Human Gaze (Surprisingly Well): Even without explicit human gaze supervision, the AI model's attention maps closely align with human eye fixations, especially for videos that are highly memorable. This suggests that the visual features driving memorability are consistently identified by both humans and the AI.
- Focus on "Things" Over "Stuff": Both humans and the AI model tend to pay more attention to distinct objects ("things") and less to amorphous background regions ("stuff"), mirroring their relative importance for memorability rather than just their pixel prevalence.
- Early Frames Matter: The AI model, trained only on memorability scores, exhibits a temporal attention bias towards the beginning of videos, mirroring human attention patterns where initial moments often capture more focus. This is observed even when the model is not explicitly biased towards temporal order.
- Dataset Quality is Crucial: The paper highlights significant issues with data leakage and inconsistent ground truth in the VideoMem dataset, which can hinder reliable evaluation and model development. This underscores the importance of careful dataset curation.
Key Results
- The proposed CNN+Transformer model achieves state-of-the-art performance on video memorability prediction, matching or exceeding existing methods on Memento10K.
- Quantitative metrics (AUC-Judd, NSS, CC) show a high degree of similarity between model attention and human gaze, often approaching human-human agreement scores.
- The model's AUC-P scores (82.91 ±1.65 on Memento10K, 88.88 ±1.29 on VideoMem) significantly exceed those of a center-bias model, indicating that the learned attention is more sophisticated than simple saliency.
- Analysis of panoptic segmentation reveals that both model and humans show lower affinity for "stuff" and higher for "things" compared to their normalized pixel counts.
Limitations and Future Work
- Dataset Size: The current datasets (10K videos) might limit generalization to the vast diversity of videos on the internet, especially in highly specific domains.
- Frame-based Processing: The model processes individual frames rather than the full video stream, potentially missing crucial temporal dynamics or continuous visual information that humans perceive.
- Future Work: The authors suggest exploring more robust datasets, investigating the impact of video complexity on alignment, and potentially incorporating full video processing techniques. They also emphasize the need for careful dataset construction and validation in future memorability research.
Summary updated