GASP: Gated Attention For Saliency Prediction

Fares Abawi, Tom Weber, Stefan Wermter

Abstract

Saliency prediction refers to the computational task of modeling overt attention. Social cues greatly influence our attention, consequently altering our eye movements and behavior. To emphasize the efficacy of such features, we present a neural model for integrating social cues and weighting their influences. Our model consists of two stages. During the first stage, we detect two social cues by following gaze, estimating gaze direction, and recognizing affect. These features are then transformed into spatiotemporal maps through image processing operations. The transformed representations are propagated to the second stage (GASP) where we explore various techniques of late fusion for integrating social cues and introduce two sub-networks for directing attention to relevant stimuli. Our experiments indicate that fusion approaches achieve better results for static integration methods, whereas non-fusion approaches for which the influence of each modality is unknown, result in better outcomes when coupled with recurrent models for dynamic saliency prediction. We show that gaze direction and affective representations contribute a prediction to ground-truth correspondence improvement of at least 5% compared to dynamic saliency models without social cues. Furthermore, affective representations improve GASP, supporting the necessity of considering affect-biased attention in predicting saliency.

Introduction

Saliency prediction—the task of predicting which parts of a visual scene will attract human attention—has traditionally relied on "bottom-up" features like color, intensity, and contrast. However, human attention is deeply social; we are naturally drawn to faces, emotional expressions, and the direction of others' gazes. Most state-of-the-art models remain "socially agnostic," failing to capture these high-level intents. GASP (Gated Attention for Saliency Prediction) addresses this gap by explicitly integrating social cues into a dynamic saliency framework.

Methodology

The GASP framework utilizes a two-stage pipeline designed to extract and then intelligently fuse social information with standard visual saliency.

The two-stage GASP pipeline: Social Cue Detection (left) extracts priority maps for gaze and affect, which are then fused via the Directed Attention Module and Sequential Integration (right).

Stage 1: Social Cue Detection (SCD)

In the first stage, the model extracts high-level features from video frames using specialized pre-trained detectors for Gaze Following (GF), Gaze Estimation (GE), and Facial Expression Recognition (FER). These are combined with a baseline Audiovisual Saliency (SP) map. These features are transformed into spatiotemporal "priority maps" that represent the social importance of different regions in the frame.

Stage 2: GASP Integration and the Directed Attention Module

The core novelty of GASP lies in its Directed Attention Module (DAM). A common problem in multimodal fusion is "modality bias," where a dominant signal (like the baseline saliency map) drowns out subtler social cues. To combat this, the authors introduced an Inverted Stream alongside the standard Direct Stream.

The Directed Attention Module (DAM) architecture, featuring the Direct and Inverted Streams. The Inverted Stream uses a surprise-based mechanism to ensure sparse social signals are not suppressed.

This Inverted Stream calculates which features are "surprising" or unexpected relative to the baseline. By weighting these informative social signals more heavily, the model ensures that sparse but vital cues—such as a small face expressing a strong emotion—are prioritized in the final prediction.

Key Results and Insights

Experimental results on the AVE (Audio-Visual Event) dataset demonstrate the efficacy of the GASP approach:

  • Performance Improvement: GASP achieved a 5% improvement in ground-truth correspondence compared to dynamic models lacking social cues. It consistently improved the performance of existing top-tier models like STAViS and UNISAL when used as a plug-in module.
  • The Role of Affect: Facial Expression Recognition (FER) was found to be a critical component, confirming the theory of "affect-biased attention"—that humans prioritize emotional content when scanning a scene.
  • Temporal Dynamics: The model performed best with a temporal context of 8 to 10 frames, suggesting that human saliency is driven by immediate social interactions rather than long-term history.
  • Reducing Central Bias: Unlike many models that default to predicting the center of the screen, GASP accurately predicted fixations on actors located at the periphery, provided they exhibited strong social cues.

Conclusion

GASP demonstrates that high-level social cues are essential for accurate saliency prediction in human-centric environments. By introducing the Directed Attention Module and its Inverted Stream, the framework successfully integrates sparse social signals that traditional models overlook. Future work may explore end-to-end training to reduce the computational overhead of the current multi-stage detector pipeline.

Summary updated

Abstract

Saliency prediction refers to the computational task of modeling overt attention. Social cues greatly influence our attention, consequently altering our eye movements and behavior. To emphasize the efficacy of such features, we present a neural model for integrating social cues and weighting their influences. Our model consists of two stages. During the first stage, we detect two social cues by following gaze, estimating gaze direction, and recognizing affect. These features are then transformed into spatiotemporal maps through image processing operations. The transformed representations are propagated to the second stage (GASP) where we explore various techniques of late fusion for integrating social cues and introduce two sub-networks for directing attention to relevant stimuli. Our experiments indicate that fusion approaches achieve better results for static integration methods, whereas non-fusion approaches for which the influence of each modality is unknown, result in better outcomes when coupled with recurrent models for dynamic saliency prediction. We show that gaze direction and affective representations contribute a prediction to ground-truth correspondence improvement of at least 5% compared to dynamic saliency models without social cues. Furthermore, affective representations improve GASP, supporting the necessity of considering affect-biased attention in predicting saliency.
BibTeX
@inproceedings{abawi2021gasp,
  author = {Fares Abawi and Tom Weber and Stefan Wermter},
  title = {{GASP: Gated Attention For Saliency Prediction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/81},
}
Figure