Introduction
Visual Sound Source Localization (VSSL) is the task of highlighting specific pixels in a video frame responsible for an emitted sound. While current models perform well on single-source scenarios, they struggle with "mixtures" where multiple objects may be sounding simultaneously. The primary challenge is disentanglement: audio in a mixture is an overlapping waveform, and visual scenes often contain multiple objects, some of which are silent. Traditional methods relying on direct audio-visual correspondence often fail because they lack access to clean, individual audio tracks during training.
Methodology: Text-Guided Disentanglement
The proposed T-VSL framework introduces text as a third modality to serve as a "semantic bridge." By leveraging tri-modal embeddings from a pre-trained AudioCLIP model, T-VSL disentangles messy audio-visual signals through a multi-stage process.
The framework operates in three key stages:
- Audio-Visual Class Instance Detection: The model first identifies "what" is making noise using AudioCLIP to predict classes present in the mixture.
- Text-Guided Disentanglement: Identified class labels (e.g., "dog", "guitar") are converted to text embeddings. These act as filters in Conditioning Blocks to extract class-specific features from the noisy audio and visual streams.
- Audio-Visual Correspondence (AVC) Block: Cross-attention aligns specific visual patches with filtered audio features to generate precise heatmaps for each source.
Experimental Setup
Evaluation was conducted on several standard benchmarks, including MUSIC (Solo and Duet), VGGSound-Instruments, and VGGSound (Single and Duet). Synthetic mixtures were created from single-source videos to provide ground truth for evaluation in multi-source scenarios.
Results and Discussion
T-VSL consistently outperforms existing state-of-the-art (SOTA) methods across all datasets. On VGGSound-Single, it achieved 53.7% [email protected], surpassing previous benchmarks like Alignment (50.8%) and AVGN (49.6%). More importantly, on the challenging VGGSound-Duet multi-source task, it reached 40.1% [email protected], a significant improvement over previous multi-source methods.
The model's success stems from using text as a "clean" anchor. Unlike mixed audio or cluttered video, text labels are semantically discrete. This allows the model to "look for" specific features. Furthermore, the two-stage approach—detecting what is in the scene before where it is—effectively reduces the impact of background noise and silent visual distractors. Quantitative analysis also shows robustness to increasing complexity; when tested with four simultaneous sources, T-VSL maintained a CIoU of 29.5%, significantly higher than competing models.
Conclusion
T-VSL demonstrates the power of tri-modal learning for complex localization tasks. While it shows remarkable zero-shot transferability and robustness to increasing source counts, it remains dependent on the quality of pre-trained encoders and may face challenges with instance ambiguity (e.g., two identical violins). Future work will explore more complex learnable text prompts to further refine the disentanglement process.
Summary updated