Abstract
Human visual attention is fundamentally social, prioritized by faces, emotional expressions, and gaze directions. Traditional saliency models often overlook these "top-down" social cues in favor of "bottom-up" features like contrast and motion. This paper introduces GASP (Gated Attention for Saliency Prediction), a framework that explicitly integrates social cues—gaze following, gaze direction, and facial affect—using a novel gated attention mechanism. By employing a "surprisal" based Directed Attention Module (DAM), GASP ensures that subtle social signals are effectively captured alongside dominant visual features. Evaluation on the AVE dataset demonstrates a significant performance boost over state-of-the-art models, highlighting the importance of social logic in human attention.
Introduction
Predicting where a person will look in a video is a complex task. While low-level features like high contrast or rapid motion are significant, humans are inherently social beings. Our attention is naturally captured by others: where they are looking (gaze following) and how they are feeling (affect-biased attention).
The challenge in computational saliency is twofold: effectively integrating diverse social data with general visual motion, and overcoming "modality bias," where strong baseline predictors cause the model to ignore subtle social cues. GASP addresses these challenges by providing social features a "fair hearing" through a specialized architecture.
Methodology
The GASP framework operates in a two-stage pipeline:
1. Social Cue Detection (SCD)
The model first extracts four distinct feature maps from video frames:
- Gaze Following (GF): Identifies where individuals in the frame are looking.
- Gaze Estimation (GE): Determines the 3D orientation of eyes.
- Facial Expression Recognition (FER): Highlights expressive facial regions using Grad-CAM.
- Saliency Prediction (SP): Provides a baseline audiovisual saliency map.
2. Multimodal Fusion with GASP
To fuse these maps, the authors introduce the Directed Attention Module (DAM) and the Gated Multimodal Unit (GMU). The DAM utilizes a "surprisal" mechanism to boost channels with unexpected contributions, ensuring that sparse social cues are not drowned out by the strong baseline saliency map. The model further utilizes the LARGMU (Late Attentive Recurrent GMU) to manage the temporal flow of video data.
Key Insights
- The Power of Surprisal: The DAM's "Inverted Stream" intensifies non-salient regions that contribute to predictions. This allows the model to prioritize subtle cues, such as a small face in a corner looking at an object, which standard models might ignore.
- Affective Saliency: The inclusion of facial expressions (FER) consistently improved model performance, providing computational evidence for "affect-biased attention"—the natural human tendency to be drawn to emotional content.
- Resistance to Center Bias: Unlike many models that default to the center of the screen, GASP follows the social logic of the scene, such as tracking an actor on the periphery because other characters are looking at them.
Results
GASP achieved state-of-the-art performance on the AVE (Audio-Visual Event) dataset, a composite of the DIEM and Coutrot databases. Incorporating gaze and affect improved correspondence with ground-truth eye-tracking data by at least 5%. The DAM + LARGMU variant reached a Normalized Scanpath Saliency (NSS) of 2.87, significantly outperforming the baseline DAVE model (2.65).
Ablation studies revealed that removing the Gaze Following (GF) component caused the most significant performance drop, confirming that "following the eyes of others" is a primary driver of human attention.
Conclusion
GASP demonstrates that explicitly modeling social cues significantly enhances video saliency prediction. While the current multi-model pipeline introduces computational overhead and depends heavily on face detection accuracy, it opens new avenues for "socially intelligent" computer vision. Future work will focus on end-to-end efficiency and deeper integration of social modalities.
Summary updated