TL;DR
TACO is a training-free framework for sound-prompted visual segmentation that identifies which visual objects are making sounds using frozen, off-the-shelf foundation models. By using a semantically constrained Non-negative Matrix Co-Factorization (Sem co-NMF) over pre-trained CLIP and CLAP features, TACO discovers shared audio-visual concepts at inference time and feeds them to an open-vocabulary segmenter (FC-CLIP), outperforming previous unsupervised, training-based methods.
What Problem Does This Paper Address?
Sound-prompted segmentation (often termed audio-visual segmentation) requires segmenting regions in an image or video frame that correspond to sounds heard in the audio track.
Most existing unsupervised methods train complex alignment modules or fine-tune foundation models using contrastive learning. However, fine-tuning requires large multimodal training sets, incurs heavy compute costs, and frequently degrades the zero-shot generalization capabilities of the base models. The authors ask: Can we achieve accurate sound-prompted segmentation without any training or fine-tuning, relying purely on the frozen representations of audio and vision backbones?
Methods
TACO reformulates cross-modal grounding as an inference-time matrix decomposition problem:
[Image] -> Frozen CLIP -> XI \
-> Sem co-NMF -> Sounding Factor (V_I) -> Frozen FC-CLIP -> Final Mask
[Audio] -> Frozen CLAP -> XA / (via Word Bank)
- Feature Extraction: Spatial visual features $X_I$ are extracted using a frozen OpenCLIP ConvNeXt backbone, while temporal audio tokens $X_A$ are extracted with frozen MSCLAP.
- Semantically Constrained Co-NMF (Sem co-NMF):
- Decomposes audio and visual matrices into activation matrices ($U_A, U_I$, constrained to $[0, 1]$ via a sigmoid function) and factor matrices ($V_A, V_I$).
- Because CLIP and CLAP inhabit disparate embedding spaces, they cannot be directly aligned via $L_1$ or $L_2$ penalties. TACO bridges them using a predefined word bank (e.g., AudioSet tags) encoded through both CLIP and CLAP text encoders to serve as semantic anchors.
- Soft-masked representations of the features are compared against these anchors to create semantic descriptors ($D_{I,k}$ and $D_{A,k}$).
- The optimization minimizes reconstruction error alongside a cross-entropy penalty between the closest audio and image semantic descriptors: $\beta_p \min_k \text{CE}(D_{I,k}, D_{A,k})$.
- Temporal Consistency Regularization: For video streams, an additional regularization term enforces cosine similarity across consecutive frames for the active sounding factors.
- Decoder Prompting: The identified sounding factor ($V_I^{k^*}$) is passed directly as a prompt to the lightweight decoder of FC-CLIP, a pre-trained open-vocabulary segmentation model, generating fine segmentation boundaries.
Datasets
The authors evaluate TACO across five datasets:
- AVS-Bench: Standard audio-visual segmentation benchmark evaluated on both Single-Source (S4, ~5K samples) and Multi-Source (MS3, 400 samples) splits.
- ADE Sound Prompted (ADE SP): Combines ADE20k images/masks with VGGSound audio.
- ADE Sound Prompted Semantic: A newly introduced semantic benchmark created by assigning category labels to the ADE SP masks.
- AVSS: A large-scale semantic audio-visual benchmark (~11K samples) with single and multi-source sounding objects.
Key Insights and Ideas
- Frozen representations already know where objects are: Separately trained image and audio foundation models preserve enough spatial and spectro-temporal information that explicit cross-modal fine-tuning is unnecessary.
- Bridging spaces via text anchors: Instead of training a projection network between audio and vision spaces, projecting both onto identical text tokens creates a shared semantic coordinate system on the fly.
- Unified multi-source representation: Rather than requiring prior knowledge of how many objects are making sounds, TACO’s sounding factor naturally aggregates all simultaneous sound-producing entities into a composite concept vector.
- Resilience to negative audio: When an audio track does not correspond to visual elements (e.g., background music overlaid on a video), the factor decomposes into noise, and the open-vocabulary segmenter correctly predicts an empty mask.
Key Results
- AVS-Bench (Binary Segmentation):
- S4 (Single Source): TACO achieves 64.04% mask-IoU and 71.50% F-score, outperforming the previous unsupervised training-based state of the art (ACL-SSL at 59.76% mask-IoU) and the training-free baseline (Box-Prompt at 51.2% mask-IoU).
- MS3 (Multi Source): Achieves 43.15% mask-IoU, beating ACL-SSL (41.08%) and Box-Prompt (41.8%), despite Box-Prompt requiring oracle knowledge of the number of sources and using SAM.
- ADE Sound Prompted: Attains 51.57% mIoU and 60.01% mAP, more than doubling the performance of trained audio-visual baselines such as DenseAV (25.5% mIoU).
- Semantic Segmentation: Outperforms the Audio-prompted FC-CLIP baseline by +8.5% mIoU on AVSS (20.50% vs. 11.95%) and +16.8% mIoU on ADE SP Semantic (37.70% vs. 20.91%).
- Efficiency: Optimization converges within ~1,000 steps, adding only ~20% computational runtime over simple forward-pass baselines while using fewer total encoder MACs than competing pipelines.
Limitations and Future Work
- Per-Sample Optimization Cost: While training-free, running iterative gradient descent (1,000–1,800 steps) at inference time introduces latency that may hinder real-time deployment.
- Multi-Class Semantic Decoding: When multiple distinct sounding classes appear together, TACO maps the entire sounding region to a single dominant class vector, leading to mislabeled categories on multi-class benchmarks like AVSS.
- Audio Backbone Reliance: Performance remains bounded by the capacity of the CLAP encoder; faint or secondary audio events missed by CLAP cannot be recovered during factorization.
Summary updated