TL;DR
BiSID-AVS is a new framework that uses binaural audio to solve the "which one is sounding?" problem in audio-visual segmentation. By leveraging spatial cues like interaural time and level differences, it can pinpoint the exact sounding instance among multiple identical-looking candidates, even in static images where traditional motion-based methods fail.
What Problem Does This Paper Address?
Imagine a scene with three identical smartphones lying on a desk, but only one is ringing. Standard Audio-Visual Segmentation (AVS) models are excellent at identifying what is sounding (a phone), but they often struggle with which one. Because they rely primarily on monaural (single-channel) audio semantics, they tend to highlight every object of the detected category, leading to "false activations" on silent distractors.
While Audio-Visual Instance Segmentation (AVIS) attempts to solve this, existing methods rely heavily on temporal cues—watching for motion or tracking objects across multiple video frames. This approach breaks down in "sparse observation" scenarios: when you only have a single snapshot, or when the sounding object (like a ringing phone or a ticking clock) doesn't move visibly.
Methods
The authors propose BiSID-AVS (Binaural Sounding Instance Disambiguation), a progressive framework built on the Segment Anything Model (SAM). The core idea is to treat binaural audio not just as a semantic label, but as a spatial sensor. The model operates in three distinct stages:
- Audio-Prompted Candidate Grounding: The model uses the semantic content of the audio (extracted via a VGGish encoder) to "prompt" the visual encoder. This identifies all potential candidates in the image that match the sound category.
- Binaural-Guided Visual Spatial Response Modeling: This stage processes the binaural audio (left and right channels) to extract spatial features like Interaural Time Differences (ITD) and Interaural Level Differences (ILD). These features are projected into the image plane to create a spatial response map, essentially a heatmap of where the sound is coming from relative to the camera.
- Spatially Conditioned Sounding Instance Refinement: Finally, the model uses "dual-level spatial conditioning." The spatial response map modulates the visual features and acts as a dense prompt for the SAM decoder. This ensures that the final segmentation mask is refined to cover only the specific instance that aligns with the binaural spatial cues.
Datasets
To evaluate this task, the researchers introduced/extended two key benchmarks:
- Binaural AVS-Synthetic: An extension of the popular AVS-Synthetic dataset, now featuring instance-level annotations and synthesized binaural audio for 31,045 samples across 46 categories.
- Habitat-AVS: A more challenging dataset created using the Habitat simulator and SoundSpaces. It features realistic 3D indoor acoustics and robotic viewpoints, specifically designed to test geometry-consistent disambiguation.
Key Insights and Ideas
- Binaural Audio as a Spatial Constraint: The most significant "aha moment" is that binaural audio provides spatial evidence that is perfectly complementary to visual information. While a single image can't tell you which phone is ringing, the slight difference in how the sound hits two microphones can.
- Moving Beyond Post-Hoc Matching: Instead of generating masks first and then trying to pick the best one (a "decision-level fusion" approach), BiSID-AVS integrates spatial cues directly into the mask formation process. This leads to much more accurate contours, especially for small or adjacent objects.
- Independence from Motion: By relying on binaural cues rather than temporal ones, the model is robust in static scenes. This makes it particularly useful for robotics, where a robot might need to identify a sound source from a single glance.
Key Results
BiSID-AVS demonstrates a massive performance leap over existing state-of-the-art AVS and AVIS methods, particularly in scenes with multiple same-class candidates:
- Quantitative Gains: On multi-candidate subsets, it improved instance-level mIoU by 35.19 percentage points on AVS-Synthetic and 41.21 points on Habitat-AVS.
- Distractor Suppression: The model significantly reduced the
F_samemetric (false activation of same-class distractors), proving it successfully ignores silent objects that look identical to the source. - Qualitative Superiority: As shown in Figure 4, while other models often "bleed" their masks across multiple objects, BiSID-AVS consistently pinpoints the correct instance, even when objects are small or tightly packed.
Limitations and Future Work
The primary limitation acknowledged by the authors is the single active source assumption. The current formulation assumes only one object is making noise at a time. In real-world "cocktail party" environments with overlapping acoustic events (e.g., a dog barking while a vacuum runs), the problem becomes significantly more complex. The authors suggest that future work will focus on multi-source sounding-instance segmentation under these overlapping conditions.
Summary updated