Which One Is Sounding? Binaural Sounding Instance Disambiguation for Audio-Visual Segmentation

Haoze Ma, Yidi Li, Yu Ma, Pengyuan Zhao, Yi Shi, Bin Ren, Yanwen Guo, Dimitrios KanoulasarXiv 2026 · arXiv

TL;DR

BiSID-AVS is a new framework that uses binaural audio to solve the "which one is sounding?" problem in audio-visual segmentation. By leveraging spatial cues like interaural time and level differences, it can pinpoint the exact sounding instance among multiple identical-looking candidates, even in static images where traditional motion-based methods fail.

What Problem Does This Paper Address?

Imagine a scene with three identical smartphones lying on a desk, but only one is ringing. Standard Audio-Visual Segmentation (AVS) models are excellent at identifying what is sounding (a phone), but they often struggle with which one. Because they rely primarily on monaural (single-channel) audio semantics, they tend to highlight every object of the detected category, leading to "false activations" on silent distractors.

While Audio-Visual Instance Segmentation (AVIS) attempts to solve this, existing methods rely heavily on temporal cues—watching for motion or tracking objects across multiple video frames. This approach breaks down in "sparse observation" scenarios: when you only have a single snapshot, or when the sounding object (like a ringing phone or a ticking clock) doesn't move visibly.

Figure 1: Task motivation of BiSID-AVS. AVS may activate multiple same-class candidates, while existing AVIS mainly relies on temporal context. BiSID-AVS instead uses binaural spatial cues to identify the true sounding instance.

Methods

The authors propose BiSID-AVS (Binaural Sounding Instance Disambiguation), a progressive framework built on the Segment Anything Model (SAM). The core idea is to treat binaural audio not just as a semantic label, but as a spatial sensor. The model operates in three distinct stages:

  1. Audio-Prompted Candidate Grounding: The model uses the semantic content of the audio (extracted via a VGGish encoder) to "prompt" the visual encoder. This identifies all potential candidates in the image that match the sound category.
  2. Binaural-Guided Visual Spatial Response Modeling: This stage processes the binaural audio (left and right channels) to extract spatial features like Interaural Time Differences (ITD) and Interaural Level Differences (ILD). These features are projected into the image plane to create a spatial response map, essentially a heatmap of where the sound is coming from relative to the camera.
  3. Spatially Conditioned Sounding Instance Refinement: Finally, the model uses "dual-level spatial conditioning." The spatial response map modulates the visual features and acts as a dense prompt for the SAM decoder. This ensures that the final segmentation mask is refined to cover only the specific instance that aligns with the binaural spatial cues.
Figure 3: Overall framework of BiSID-AVS. The model progressively performs (A) category-consistent candidate grounding, (B) binaural-guided image-plane spatial response modeling, and (C) sounding-instance mask refinement. Spatial evidence conditions both candidate-feature formation and mask decoding to refine category-consistent regions into the true sounding-instance mask.

Datasets

To evaluate this task, the researchers introduced/extended two key benchmarks:

  • Binaural AVS-Synthetic: An extension of the popular AVS-Synthetic dataset, now featuring instance-level annotations and synthesized binaural audio for 31,045 samples across 46 categories.
  • Habitat-AVS: A more challenging dataset created using the Habitat simulator and SoundSpaces. It features realistic 3D indoor acoustics and robotic viewpoints, specifically designed to test geometry-consistent disambiguation.

Key Insights and Ideas

  • Binaural Audio as a Spatial Constraint: The most significant "aha moment" is that binaural audio provides spatial evidence that is perfectly complementary to visual information. While a single image can't tell you which phone is ringing, the slight difference in how the sound hits two microphones can.
  • Moving Beyond Post-Hoc Matching: Instead of generating masks first and then trying to pick the best one (a "decision-level fusion" approach), BiSID-AVS integrates spatial cues directly into the mask formation process. This leads to much more accurate contours, especially for small or adjacent objects.
  • Independence from Motion: By relying on binaural cues rather than temporal ones, the model is robust in static scenes. This makes it particularly useful for robotics, where a robot might need to identify a sound source from a single glance.

Key Results

BiSID-AVS demonstrates a massive performance leap over existing state-of-the-art AVS and AVIS methods, particularly in scenes with multiple same-class candidates:

  • Quantitative Gains: On multi-candidate subsets, it improved instance-level mIoU by 35.19 percentage points on AVS-Synthetic and 41.21 points on Habitat-AVS.
  • Distractor Suppression: The model significantly reduced the F_same metric (false activation of same-class distractors), proving it successfully ignores silent objects that look identical to the source.
  • Qualitative Superiority: As shown in Figure 4, while other models often "bleed" their masks across multiple objects, BiSID-AVS consistently pinpoints the correct instance, even when objects are small or tightly packed.
Figure 4: Qualitative comparison on representative multi- same-class-candidate scenes on two datasets.

Limitations and Future Work

The primary limitation acknowledged by the authors is the single active source assumption. The current formulation assumes only one object is making noise at a time. In real-world "cocktail party" environments with overlapping acoustic events (e.g., a dog barking while a vacuum runs), the problem becomes significantly more complex. The authors suggest that future work will focus on multi-source sounding-instance segmentation under these overlapping conditions.

Summary updated

Abstract

Existing audio-visual segmentation methods mainly predict sound-related regions or category masks and often fail to identify the sounding instance when multiple same-class candidates coexist. Recent audio-visual instance segmentation methods address instance-level prediction but rely largely on motion and cross-frame association, which may become unreliable under sparse observations or inconspicuous motion. To this end, we introduce binaural audio, whose interaural differences encode source-direction information, to provide spatial constraints for distinguishing the sounding instance among same-class visual candidates. We propose Binaural Sounding Instance Disambiguation for Audio-Visual Segmentation (BiSID-AVS), which progressively refines category-level candidate responses into a mask of the true sounding instance. BiSID-AVS first grounds category-consistent candidate regions through audio prompting, then projects binaural cues into an image-plane spatial response map, and finally refines the true sounding instance through dual-level spatial conditioning at the feature and decoder levels. Experiments on the binaurally extended AVS-Synthetic benchmark and Habitat-AVS dataset for robotic indoor perception show that BiSID-AVS achieves the best performance on the multi-candidate subsets, improving instance-level mIoU by 35.19 and 41.21 percentage points, respectively, while substantially reducing false activation on non-sounding same-class regions.
BibTeX
@misc{ma2026which,
  author = {Haoze Ma and Yidi Li and Yu Ma and Pengyuan Zhao and Yi Shi and Bin Ren and Yanwen Guo and Dimitrios Kanoulas},
  title = {{Which One Is Sounding? Binaural Sounding Instance Disambiguation for Audio-Visual Segmentation}},
  year = {2026},
  eprint = {2611.06074},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06074},
}
Figure