Audio Visual Saliency Prediction

48 concepts39 papers1 proposal

For decades, computational models of visual saliency have treated human vision as an isolated system. These models predict where a person will look by analyzing pixel-level features like color contrast, edge orientation, and localized motion. For instance, Teng Li demonstrates how biological factors like color prioritization can be modeled to predict gaze in static images (Bio-driven visual saliency detection with color factor).

However, the real world is rarely silent. When we watch a video, converse in a crowded room, or explore a virtual environment, our eyes do not move in a sensory vacuum. Traditional visual-only models suffer from a fundamental limitation: they ignore the auditory triggers that constantly redirect human attention. If an object breaks off-screen or a person begins to speak, our gaze shifts instantly—long before any visual feature changes within our current field of view. As noted by Antigoni Tsiami et al., relying exclusively on visual features creates a significant gap between predicted saliency maps and actual human gaze patterns (STAViS: Spatio-Temporal AudioVisual Saliency Network).

Sound as a Spatial Pointer

The core intuition behind audio-visual saliency is that auditory cues act as an early-warning system and a spatial pointer for the visual cortex. Sound provides a 360-degree awareness that vision lacks. This cross-modal guidance is particularly powerful in two scenarios: social interactions and immersive environments.

Conversational Scenarios and Active Speakers

In environments featuring multiple people, human attention is heavily governed by social heuristics—primarily, we look at whoever is talking. Visual-only models often get distracted by sudden movements or central positioning. Yufan Liu et al. discovered that while motion drives attention in silent environments, it becomes a secondary cue when audio is introduced (Learning to Predict Salient Faces: A Novel Visual-Audio Saliency Model). Auditory information dominates motion-based saliency, rapidly pulling the viewer's gaze toward the active speaker.

To capture this behavior, Yuqi Hou et al. developed frameworks that leverage the explicit correlation between speech and lip movements to track gaze in conversational settings (Multi-Modal Gaze Following in Conversational Scenarios). Similarly, Fiona Ryan et al. demonstrated that combining spatial audio with visual cues like head orientation resolves ambiguities in complex, multi-speaker acoustic scenes (Egocentric Auditory Attention Localization in Conversations).

Immersive and Omnidirectional Environments

The necessity of audio becomes even more pronounced in 360-degree or omnidirectional videos (ODVs). In virtual reality, a user can only see a fraction of the scene at any given moment. Visual-only models are fundamentally incapable of predicting when a user will turn their head to look behind them.

Yuxin Zhu et al. showed that spatial audio (ambisonics) significantly expands a viewer's exploration of a virtual environment, serving as a critical mechanism to guide gaze toward events occurring outside the immediate field of view (Audio-visual Saliency for Omnidirectional Videos). Without integrating these spatialized auditory features, forecasting models fail to capture sudden, audio-driven head rotations (Gaze Prediction in Dynamic 360° Immersive Videos).

Architectures for Cross-Modal Fusion

To build models that mimic this human capability, researchers have moved away from simple "late fusion" (where visual and audio saliency maps are computed independently and averaged) toward deep, collaborative architectures that bind the modalities together early in the processing pipeline.

The STAViS network proposed by Antigoni Tsiami et al. combines spatio-temporal visual features with raw audio processing in a single end-to-end network (STAViS: Spatio-Temporal AudioVisual Saliency Network). By fusing these streams early, the network allows auditory features to directly modulate the visual feature maps, highlighting regions that correspond to active sound sources before the final saliency map is generated.

Another approach involves cross-modal attention mechanisms. Instead of merging features statically, models can use information from one modality to dynamically "spotlight" relevant features in another. For example, Seunghyun Yoon et al. introduced an "Attentive Modality Hopping" mechanism (Attentive Modality Hopping Mechanism for Speech Emotion Recognition). Conditioning the attention of one data stream on another allows the model to prioritize segments where visual and auditory signals reinforce each other.

Similarly, Hailong Ning et al. utilized audio-visual collaborative representation learning to ensure that the model learns shared representations of dynamic events, directly mapping what is moving to what is making noise (Audio-Visual Collaborative Representation Learning for Dynamic Saliency Prediction).

By treating sound not as an afterthought but as an intrinsic driver of visual orientation, audio-visual saliency models achieve a much closer approximation of human behavior. They succeed because they respect the biology of attention: vision tells us what details are in front of us, but audio often tells us where to look next.

Key Foundational Work

Guest Editorial Behavior Understanding and Developmental Robotics (2014) — contributed 1 concept elements to this direction.
Interference of Speech and Interior Noise of Chinese High-Speed Trains with Task Performance (2012) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_modal_saliency_eb595499,
  title = {cross-modal saliency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modal-saliency/rdr_eb59549961bd404390062fa951540453},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_modal_saliency_eb595499_v1,
  title = {cross-modal saliency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modal-saliency/rdr_eb59549961bd404390062fa951540453/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake