TACO: Training-free Sound Prompted Segmentation via Deep Audio-visual CO-factorization

Hugo Malard, Michel Olvera, Stéphane Lathuilière, Slim Essid

Abstract

Sound-prompted segmentation aims to identify the specific pixels in an image responsible for a given audio signal. Traditional methods rely on extensive fine-tuning or training complex adapter modules, which can degrade the general knowledge of pre-trained models and require large-scale aligned datasets. We present TACO (Training-free Audio-visual CO-factorization), a framework that achieves precise audio-visual grounding using only frozen representations. By employing Semantically Constrained Non-negative Matrix Factorization (Sem co-NMF), TACO decomposes audio and visual features into shared semantic factors. Our approach sets new state-of-the-art benchmarks for unsupervised audio-visual segmentation while maintaining high interpretability and robustness.

Introduction

In audio-visual perception, identifying which part of an image is making a sound—known as sound-prompted segmentation—is a fundamental challenge. Current standard approaches typically involve fine-tuning massive pre-trained models like CLIP on paired audio-visual data. However, this often leads to a loss of generalization and faces difficulties with local alignment, as most video data only provides global labels. TACO addresses these issues by asking whether precise grounding can be achieved using frozen models without any additional training.

Methodology

TACO introduces a training-free pipeline centered on Semantically Constrained Non-negative Matrix Factorization (Sem co-NMF). The framework utilizes frozen backbones: CLIP for visual features and CLAP for audio features. While these models are pre-aligned with text, they are not inherently aligned with each other at a local level.

Overview of the TACO framework for joint audio-visual deep feature decomposition.

To bridge this gap, TACO uses a "word bank" of semantic anchors to project both modalities into a shared text-based space. During inference, the system performs co-factorization to decompose features into shared semantic factors and their spatial/temporal activations. The identified "sounding factor" then prompts FC-CLIP, a pre-trained open-vocabulary segmenter, to generate high-quality masks. For video sequences, temporal consistency is maintained through a regularization term that prevents flickering.

Datasets and Evaluation

The framework was evaluated on several key benchmarks:

  • AVS-Bench (S4 and MS3): Standard datasets for single and multi-source audio-visual segmentation.
  • ADE20k sound-prompted: A dataset pairing ADE20k images with sounds from VGGSound.
  • AVSS (Audio-Visual Semantic Segmentation): Requires both segmentation and class labeling.
  • ADE Sound Prompted Semantic: A new benchmark introduced by the authors to test semantic segmentation accuracy.

Key Results and Insights

TACO significantly outperforms previous unsupervised methods across multiple benchmarks. On AVS-Bench S4, it achieved a 64.04 mask-IoU, surpassing the previous leader, ACL-SSL. On the ADE SP dataset, TACO reached 51.57 mIoU, nearly doubling the performance of the DenseAV model.

Performance comparison of TACO against state-of-the-art methods on AVS-Bench S4 and MS3 datasets.

Beyond quantitative metrics, TACO offers several qualitative advantages:

  • Interpretability: The NMF decomposition allows researchers to visualize intermediate semantic factors, such as identifying a "baby babbling" concept in both modalities.
  • Robustness: The model is resilient to "negative" audio (irrelevant background noise), producing empty masks when no shared semantic concept exists.
  • Multi-Source Capability: TACO can simultaneously capture multiple objects if they are semantically related to the audio prompt.

Conclusion

TACO demonstrates that the latent spaces of frozen models like CLIP and CLAP already contain the necessary information for complex cross-modal grounding. While the optimization step at inference adds approximately 20% compute overhead and performance is somewhat dependent on the quality of the audio encoder and word bank, TACO provides a powerful, interpretable, and training-free alternative for audio-visual tasks. Future work may focus on dynamic semantic anchor generation and improving inference speed.

Summary updated

BibTeX
@article{malard2025taco,
  author = {Hugo Malard and Michel Olvera and Stéphane Lathuilière and Slim Essid},
  title = {{TACO: Training-free Sound Prompted Segmentation via Deep Audio-visual CO-factorization}},
  journal = {Accepted by TMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=Xt9sdzQQlJ},
}
Figure