T-VSL: Text-Guided Visual Sound Source Localization in Mixtures

Tanvir Mahmud, Yapeng Tian, Diana Marculescu

Introduction

Visual Sound Source Localization (VSSL) is the task of highlighting specific pixels in a video frame responsible for an emitted sound. While current models perform well on single-source scenarios, they struggle with "mixtures" where multiple objects may be sounding simultaneously. The primary challenge is disentanglement: audio in a mixture is an overlapping waveform, and visual scenes often contain multiple objects, some of which are silent. Traditional methods relying on direct audio-visual correspondence often fail because they lack access to clean, individual audio tracks during training.

Methodology: Text-Guided Disentanglement

The proposed T-VSL framework introduces text as a third modality to serve as a "semantic bridge." By leveraging tri-modal embeddings from a pre-trained AudioCLIP model, T-VSL disentangles messy audio-visual signals through a multi-stage process.

T-VSL Architecture: The model uses text features to guide the disentanglement of audio and visual features through conditioning blocks before final localization.

The framework operates in three key stages:

  1. Audio-Visual Class Instance Detection: The model first identifies "what" is making noise using AudioCLIP to predict classes present in the mixture.
  2. Text-Guided Disentanglement: Identified class labels (e.g., "dog", "guitar") are converted to text embeddings. These act as filters in Conditioning Blocks to extract class-specific features from the noisy audio and visual streams.
  3. Audio-Visual Correspondence (AVC) Block: Cross-attention aligns specific visual patches with filtered audio features to generate precise heatmaps for each source.

Experimental Setup

Evaluation was conducted on several standard benchmarks, including MUSIC (Solo and Duet), VGGSound-Instruments, and VGGSound (Single and Duet). Synthetic mixtures were created from single-source videos to provide ground truth for evaluation in multi-source scenarios.

Results and Discussion

T-VSL consistently outperforms existing state-of-the-art (SOTA) methods across all datasets. On VGGSound-Single, it achieved 53.7% [email protected], surpassing previous benchmarks like Alignment (50.8%) and AVGN (49.6%). More importantly, on the challenging VGGSound-Duet multi-source task, it reached 40.1% [email protected], a significant improvement over previous multi-source methods.

Qualitative Comparison: T-VSL provides more accurate and localized heatmaps for individual sources in complex mixtures compared to previous SOTA methods.

The model's success stems from using text as a "clean" anchor. Unlike mixed audio or cluttered video, text labels are semantically discrete. This allows the model to "look for" specific features. Furthermore, the two-stage approach—detecting what is in the scene before where it is—effectively reduces the impact of background noise and silent visual distractors. Quantitative analysis also shows robustness to increasing complexity; when tested with four simultaneous sources, T-VSL maintained a CIoU of 29.5%, significantly higher than competing models.

Conclusion

T-VSL demonstrates the power of tri-modal learning for complex localization tasks. While it shows remarkable zero-shot transferability and robustness to increasing source counts, it remains dependent on the quality of pre-trained encoders and may face challenges with instance ambiguity (e.g., two identical violins). Future work will explore more complex learnable text prompts to further refine the disentanglement process.

Summary updated

Abstract

Visual sound source localization poses a significant challenge in identifying the semantic region of each sounding source within a video. Existing self-supervised and weakly supervised source localization methods struggle to accurately distinguish the semantic regions of each sounding object, particularly in multi-source mixtures. These methods often rely on audio-visual correspondence as guidance, which can lead to substantial performance drops in complex multi-source localization scenarios. The lack of access to individual source sounds in multi-source mixtures during training exacerbates the difficulty of learning effective audio-visual correspondence for localization. To address this limitation, in this paper, we propose incorporating the text modality as an intermediate feature guide using tri-modal joint embedding models (e.g., AudioCLIP) to disentangle the semantic audio-visual source correspondence in multi-source mixtures. Our framework, dubbed T-VSL, begins by predicting the class of sounding entities in mixtures. Subsequently, the textual representation of each sounding source is employed as guidance to disentangle fine-grained audio-visual source correspondence from multi-source mixtures, leveraging the tri-modal AudioCLIP embedding. This approach enables our framework to handle a flexible number of sources and exhibits promising zero-shot transferability to unseen classes during test time. Extensive experiments conducted on the MUSIC, VGGSound, and VGGSound-Instruments datasets demonstrate significant performance improvements over state-of-the-art methods. Code is released at https://github.com/enyac-group/T-VSL/tree/main
BibTeX
@inproceedings{mahmud2024t,
  author = {Tanvir Mahmud and Yapeng Tian and Diana Marculescu},
  title = {{T-VSL: Text-Guided Visual Sound Source Localization in Mixtures}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02525},
}
Figure