Audio-Visual Integration for Human Attention Modeling

This direction is split into other directions: cross-modal saliency. Its last version stays readable and citable here.

Version 1 of 187 papers107 conceptsfrozen 2026-10-04

Traditional computer vision models treat human attention as a purely visual phenomenon. They analyze pixels to find "saliency"—the bright colors, sharp edges, or moving objects that naturally draw the eye. However, humans do not live in a silent world. In reality, our eyes are often "slaves" to our ears. We turn our heads toward a sudden bang behind us or shift our gaze to a person the moment they start speaking, often before we even see them.

Researchers are now moving beyond vision-only models to embrace Audio-Visual Attention Modeling. This direction treats sound not just as an accompaniment to video, but as a primary driver of human intent, spatial orientation, and gaze behavior.

Sound as a Spatial Anchor

In immersive 360-degree or panoramic environments, the "field of view" is a limitation of the hardware, not the scene. A user can look anywhere. Yi Zhang 0076, Fang-Yi Chao, and Deng-Ping Fan demonstrated in ASOD60K: Audio-Induced Salient Object Detection in Panoramic Videos that without sound, human exploration of a 360-degree space is scattered and unpredictable. When spatial audio is introduced, it acts as an "anchor," creating significantly higher gaze consistency among different viewers.

To capture this, models like the one proposed by Haoyuan Cao and Bhiksha Raj in Panoramic Video Salient Object Detection with Ambisonic Audio Guidance utilize ambisonic audio—multi-channel sound that preserves 3D directional cues. By fusing these acoustic directions with spherical visual projections, these systems can "hear" where a salient object is located even if it is currently behind the camera's viewport.

Listening to Look into the Future

One of the most powerful applications of this integration is gaze anticipation: predicting where a person will look before they actually move their eyes. Vision-only models struggle here because they can only react to what is already visible.

In Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation, Bolin Lai, Fiona Ryan, and James M. Rehg show that audio provides a "temporal lead." A voice or a footstep serves as a causal trigger. By using a Contrastive Spatial-Temporal Separable (CSTS) fusion approach, their model leverages the fact that sound reaches us from all directions simultaneously, allowing the system to forecast gaze shifts up to two seconds into the future—a feat nearly impossible with visual data alone.

The Social Heuristic: Who is Speaking?

In social settings, human attention follows specific "social heuristics." We tend to look at whoever is talking. This is the core intuition behind MMGaze, a framework that improves gaze-following by identifying active speakers through lip-sync correlation. By integrating audio, the model can resolve ambiguities in crowded scenes where multiple people are visible but only one is the focus of attention.

This synergy works both ways. While audio helps us find the speaker, visual cues help us "tune in" to the right sound. Cunhang Fan and colleagues have explored this in works like DBPNet and DARNet. They found that the neural signature of auditory attention (measured via EEG) is significantly stronger when a person has both audio and visual access to a speaker. This suggests that "auditory attention" is fundamentally a multimodal process; our brains use the sight of moving lips to help isolate a specific voice in a noisy room (the "cocktail party problem").

From Pixels to Reasoning

The field is currently shifting from low-level feature fusion to high-level reasoning. Early models like DAVE by Ali Borji and Hamed R. Tavakoli focused on training deep embeddings to map sound to saliency maps. Newer frameworks, such as ViRAC by Jeonghun Hwang, use Large Language Models (LLMs) and Vision-Language Models (VLMs) to "think" about environmental triggers.

Instead of just reacting to a loud noise, these agents can reason: "I hear a car horn behind me; I should check for safety." Similarly, the EyEar framework by Xiaochuan Liu and Ruihua Song models gaze as a dynamical system influenced by "semantic attraction"—where the meaning of a spoken narration pulls the viewer's eyes toward specific objects in the scene.

Why It Works: The Physiological Reality

The success of these models is rooted in human physiology. As noted in DAVE, there is a measurable temporal lag in how humans react to auditory vs. visual stimuli. By accounting for this delay and the crossmodal "boost" that occurs when sight and sound align, researchers are creating artificial systems that don't just track pixels, but truly simulate the way humans prioritize and navigate their sensory world.

Papers in this version

Showing 60 of 87 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_audio_visual_integration_for_human_atten_6699315c,
  title = {Audio-Visual Integration for Human Attention Modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/audio-visual-integration-for-human-attention-modeling/rdr_6699315c02384c04b772ff706937a26a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_audio_visual_integration_for_human_atten_6699315c_v1,
  title = {Audio-Visual Integration for Human Attention Modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/audio-visual-integration-for-human-attention-modeling/rdr_6699315c02384c04b772ff706937a26a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}