TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization

Hugo Malard, Michel Olvera, Stéphane Lathuilière, Slim Essid

TL;DR

TACO is a training-free framework for sound-prompted visual segmentation that identifies which visual objects are making sounds using frozen, off-the-shelf foundation models. By using a semantically constrained Non-negative Matrix Co-Factorization (Sem co-NMF) over pre-trained CLIP and CLAP features, TACO discovers shared audio-visual concepts at inference time and feeds them to an open-vocabulary segmenter (FC-CLIP), outperforming previous unsupervised, training-based methods.

What Problem Does This Paper Address?

Sound-prompted segmentation (often termed audio-visual segmentation) requires segmenting regions in an image or video frame that correspond to sounds heard in the audio track.

Most existing unsupervised methods train complex alignment modules or fine-tune foundation models using contrastive learning. However, fine-tuning requires large multimodal training sets, incurs heavy compute costs, and frequently degrades the zero-shot generalization capabilities of the base models. The authors ask: Can we achieve accurate sound-prompted segmentation without any training or fine-tuning, relying purely on the frozen representations of audio and vision backbones?

Methods

TACO reformulates cross-modal grounding as an inference-time matrix decomposition problem:

[Image] -> Frozen CLIP -> XI \
                              -> Sem co-NMF -> Sounding Factor (V_I) -> Frozen FC-CLIP -> Final Mask
[Audio] -> Frozen CLAP -> XA /   (via Word Bank)
  1. Feature Extraction: Spatial visual features $X_I$ are extracted using a frozen OpenCLIP ConvNeXt backbone, while temporal audio tokens $X_A$ are extracted with frozen MSCLAP.
  2. Semantically Constrained Co-NMF (Sem co-NMF):
  3. Decomposes audio and visual matrices into activation matrices ($U_A, U_I$, constrained to $[0, 1]$ via a sigmoid function) and factor matrices ($V_A, V_I$).
  4. Because CLIP and CLAP inhabit disparate embedding spaces, they cannot be directly aligned via $L_1$ or $L_2$ penalties. TACO bridges them using a predefined word bank (e.g., AudioSet tags) encoded through both CLIP and CLAP text encoders to serve as semantic anchors.
  5. Soft-masked representations of the features are compared against these anchors to create semantic descriptors ($D_{I,k}$ and $D_{A,k}$).
  6. The optimization minimizes reconstruction error alongside a cross-entropy penalty between the closest audio and image semantic descriptors: $\beta_p \min_k \text{CE}(D_{I,k}, D_{A,k})$.
  7. Temporal Consistency Regularization: For video streams, an additional regularization term enforces cosine similarity across consecutive frames for the active sounding factors.
  8. Decoder Prompting: The identified sounding factor ($V_I^{k^*}$) is passed directly as a prompt to the lightweight decoder of FC-CLIP, a pre-trained open-vocabulary segmentation model, generating fine segmentation boundaries.

Datasets

The authors evaluate TACO across five datasets:

  • AVS-Bench: Standard audio-visual segmentation benchmark evaluated on both Single-Source (S4, ~5K samples) and Multi-Source (MS3, 400 samples) splits.
  • ADE Sound Prompted (ADE SP): Combines ADE20k images/masks with VGGSound audio.
  • ADE Sound Prompted Semantic: A newly introduced semantic benchmark created by assigning category labels to the ADE SP masks.
  • AVSS: A large-scale semantic audio-visual benchmark (~11K samples) with single and multi-source sounding objects.

Key Insights and Ideas

  • Frozen representations already know where objects are: Separately trained image and audio foundation models preserve enough spatial and spectro-temporal information that explicit cross-modal fine-tuning is unnecessary.
  • Bridging spaces via text anchors: Instead of training a projection network between audio and vision spaces, projecting both onto identical text tokens creates a shared semantic coordinate system on the fly.
  • Unified multi-source representation: Rather than requiring prior knowledge of how many objects are making sounds, TACO’s sounding factor naturally aggregates all simultaneous sound-producing entities into a composite concept vector.
  • Resilience to negative audio: When an audio track does not correspond to visual elements (e.g., background music overlaid on a video), the factor decomposes into noise, and the open-vocabulary segmenter correctly predicts an empty mask.

Key Results

  • AVS-Bench (Binary Segmentation):
  • S4 (Single Source): TACO achieves 64.04% mask-IoU and 71.50% F-score, outperforming the previous unsupervised training-based state of the art (ACL-SSL at 59.76% mask-IoU) and the training-free baseline (Box-Prompt at 51.2% mask-IoU).
  • MS3 (Multi Source): Achieves 43.15% mask-IoU, beating ACL-SSL (41.08%) and Box-Prompt (41.8%), despite Box-Prompt requiring oracle knowledge of the number of sources and using SAM.
  • ADE Sound Prompted: Attains 51.57% mIoU and 60.01% mAP, more than doubling the performance of trained audio-visual baselines such as DenseAV (25.5% mIoU).
  • Semantic Segmentation: Outperforms the Audio-prompted FC-CLIP baseline by +8.5% mIoU on AVSS (20.50% vs. 11.95%) and +16.8% mIoU on ADE SP Semantic (37.70% vs. 20.91%).
  • Efficiency: Optimization converges within ~1,000 steps, adding only ~20% computational runtime over simple forward-pass baselines while using fewer total encoder MACs than competing pipelines.

Limitations and Future Work

  • Per-Sample Optimization Cost: While training-free, running iterative gradient descent (1,000–1,800 steps) at inference time introduces latency that may hinder real-time deployment.
  • Multi-Class Semantic Decoding: When multiple distinct sounding classes appear together, TACO maps the entire sounding region to a single dominant class vector, leading to mislabeled categories on multi-class benchmarks like AVSS.
  • Audio Backbone Reliance: Performance remains bounded by the capacity of the CLAP encoder; faint or secondary audio events missed by CLAP cannot be recovered during factorization.

Summary updated

Abstract

Large-scale pre-trained audio and image models demonstrate an unprecedented degree of generalization, making them suitable for a wide range of applications. Here, we tackle the specific task of sound-prompted segmentation, aiming to segment image regions corresponding to objects heard in an audio signal. Most existing approaches tackle this problem by fine-tuning pre-trained models or by training additional modules specifically for the task. We adopt a different strategy: we introduce a training-free approach that leverages Non-negative Matrix Factorization (NMF) to co-factorize audio and visual features from pre-trained models so as to reveal shared interpretable concepts. These concepts are passed on to an open-vocabulary segmentation model for precise segmentation maps. By using frozen pre-trained models, our method achieves high generalization and establishes state-of-the-art performance in unsupervised sound-prompted segmentation, significantly surpassing previous unsupervised methods.
BibTeX
@article{malard2025taco,
  author = {Hugo Malard and Michel Olvera and Stéphane Lathuilière and Slim Essid},
  title = {{TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization}},
  journal = {Accepted by TMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=Xt9sdzQQlJ},
}
Figure