GASP: Gated Attention for Saliency Prediction

Fares Abawi, Tom Weber, Stefan Wermter

Abstract

Human visual attention is fundamentally social, prioritized by faces, emotional expressions, and gaze directions. Traditional saliency models often overlook these "top-down" social cues in favor of "bottom-up" features like contrast and motion. This paper introduces GASP (Gated Attention for Saliency Prediction), a framework that explicitly integrates social cues—gaze following, gaze direction, and facial affect—using a novel gated attention mechanism. By employing a "surprisal" based Directed Attention Module (DAM), GASP ensures that subtle social signals are effectively captured alongside dominant visual features. Evaluation on the AVE dataset demonstrates a significant performance boost over state-of-the-art models, highlighting the importance of social logic in human attention.

Introduction

Predicting where a person will look in a video is a complex task. While low-level features like high contrast or rapid motion are significant, humans are inherently social beings. Our attention is naturally captured by others: where they are looking (gaze following) and how they are feeling (affect-biased attention).

The challenge in computational saliency is twofold: effectively integrating diverse social data with general visual motion, and overcoming "modality bias," where strong baseline predictors cause the model to ignore subtle social cues. GASP addresses these challenges by providing social features a "fair hearing" through a specialized architecture.

Methodology

The GASP framework operates in a two-stage pipeline:

1. Social Cue Detection (SCD)

The model first extracts four distinct feature maps from video frames:

  • Gaze Following (GF): Identifies where individuals in the frame are looking.
  • Gaze Estimation (GE): Determines the 3D orientation of eyes.
  • Facial Expression Recognition (FER): Highlights expressive facial regions using Grad-CAM.
  • Saliency Prediction (SP): Provides a baseline audiovisual saliency map.

2. Multimodal Fusion with GASP

To fuse these maps, the authors introduce the Directed Attention Module (DAM) and the Gated Multimodal Unit (GMU). The DAM utilizes a "surprisal" mechanism to boost channels with unexpected contributions, ensuring that sparse social cues are not drowned out by the strong baseline saliency map. The model further utilizes the LARGMU (Late Attentive Recurrent GMU) to manage the temporal flow of video data.

The GASP framework architecture, illustrating the two-stage process: Social Cue Detection followed by sequential integration and gated attention.

Key Insights

  • The Power of Surprisal: The DAM's "Inverted Stream" intensifies non-salient regions that contribute to predictions. This allows the model to prioritize subtle cues, such as a small face in a corner looking at an object, which standard models might ignore.
  • Affective Saliency: The inclusion of facial expressions (FER) consistently improved model performance, providing computational evidence for "affect-biased attention"—the natural human tendency to be drawn to emotional content.
  • Resistance to Center Bias: Unlike many models that default to the center of the screen, GASP follows the social logic of the scene, such as tracking an actor on the periphery because other characters are looking at them.

Results

GASP achieved state-of-the-art performance on the AVE (Audio-Visual Event) dataset, a composite of the DIEM and Coutrot databases. Incorporating gaze and affect improved correspondence with ground-truth eye-tracking data by at least 5%. The DAM + LARGMU variant reached a Normalized Scanpath Saliency (NSS) of 2.87, significantly outperforming the baseline DAVE model (2.65).

Qualitative results comparing ground-truth saliency with GASP variants. The model successfully captures social interactions that baseline models miss.

Ablation studies revealed that removing the Gaze Following (GF) component caused the most significant performance drop, confirming that "following the eyes of others" is a primary driver of human attention.

Conclusion

GASP demonstrates that explicitly modeling social cues significantly enhances video saliency prediction. While the current multi-model pipeline introduces computational overhead and depends heavily on face detection accuracy, it opens new avenues for "socially intelligent" computer vision. Future work will focus on end-to-end efficiency and deeper integration of social modalities.

Summary updated

Abstract

Saliency prediction refers to the computational task of modeling overt attention. Social cues greatly influence our attention, consequently altering our eye movements and behavior. To emphasize the efficacy of such features, we present a neural model for integrating social cues and weighting their influences. Our model consists of two stages. During the first stage, we detect two social cues by following gaze, estimating gaze direction, and recognizing affect. These features are then transformed into spatiotemporal maps through image processing operations. The transformed representations are propagated to the second stage (GASP) where we explore various techniques of late fusion for integrating social cues and introduce two sub-networks for directing attention to relevant stimuli. Our experiments indicate that fusion approaches achieve better results for static integration methods, whereas non-fusion approaches for which the influence of each modality is unknown, result in better outcomes when coupled with recurrent models for dynamic saliency prediction. We show that gaze direction and affective representations contribute a prediction to ground-truth correspondence improvement of at least 5% compared to dynamic saliency models without social cues. Furthermore, affective representations improve GASP, supporting the necessity of considering affect-biased attention in predicting saliency.
BibTeX
@inproceedings{abawi2021gasp,
  author = {Fares Abawi and Tom Weber and Stefan Wermter},
  title = {{GASP: Gated Attention for Saliency Prediction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/81},
}
Figure