Abstract
Learning how to localize and separate individual object sounds in the audio channel of the video is a difficult task. Current state-of-the-art methods predict audio masks from artificially mixed spectrograms, known as Mix-and-Separate framework. We propose an audio-visual co-segmentation, where the network learns both what individual objects look and sound like, from videos labeled with only object labels. Unlike other recent visually-guided audio source separation frameworks, our architecture can be learned in an end-to-end manner and requires no additional supervision or bounding box proposals. Specifically, we introduce weakly-supervised object segmentation in the context of sound separation. We also formulate spectrogram mask prediction using a set of learned mask bases, which combine using coefficients conditioned on the output of object segmentation, a design that facilitates separation. Extensive experiments on the MUSIC dataset show that our proposed approach outperforms state-of-the-art methods on visually guided sound source separation and sound denoising.
Introduction
Isolating a specific sound from a noisy environment—often referred to as the "cocktail party effect"—is a natural human ability but remains a significant challenge for artificial intelligence. While recent "visually-guided" methods have made strides in this area, they often suffer from heavy reliance on supervision (e.g., pre-trained object detectors or manual bounding box annotations) or simplistic assumptions about the number of sound sources in a scene.
This paper introduces an end-to-end framework for localizing and separating individual sound sources in video using only video-level labels (e.g., "cello," "flute"). The authors tackle the challenge of weakly-supervised co-segmentation: learning what an object looks like and how it sounds simultaneously, without requiring expensive bounding box proposals.
Methods
The proposed architecture consists of two primary paths that interact to perform "Mix-and-Separate" self-supervision. In this paradigm, the model learns to recover original audio from an artificial mixture of two videos.
Video Frame Semantic Segmentation Path
Using a ResNet-18 backbone, the model generates class-specific attention maps. It employs a dual-attention strategy:
- Expansive Detector: Identifies the general object region.
- Discriminative Detector: Pinpoints the most salient parts of the object. These are combined to create a final attention map ($X_m$) used to pool visual features.
Spectrogram Mask Prediction Path
Instead of predicting a final audio mask directly, the model uses an Attention U-Net to generate $k$ "audio mask bases." This decomposition is a key innovation of the work.
Mask Coefficient Generator
The model fuses the pooled visual features with the audio features to predict a set of sparse coefficients. The final spectrogram mask is then constructed as a linear combination of the mask bases weighted by these coefficients. This bi-linear decomposition allows the network to learn a general "vocabulary" of sounds (bases) and use visual cues to "dial in" the right combination for a specific object.
Experiments and Results
The model was evaluated on the MUSIC Dataset (685 videos of musical solos and duets) and AudioSet-SingleSource (for cross-dataset evaluation).
Key Results
- State-of-the-Art Performance: On the MUSIC dataset, the proposed method achieved a Signal-to-Distortion Ratio (SDR) of 9.14, significantly outperforming the "Sound-of-Pixels" baseline (7.81) and "CO-SEPARATION" (7.64).
- High Localization Accuracy: Despite only having video-level labels, the model achieved up to 93.69% classification accuracy, proving that its internal attention maps successfully localized the correct instruments.
- Robust Generalization: In cross-dataset tests (training on MUSIC, testing on AudioSet), the model showed a nearly 10x performance increase in SDR compared to previous baselines, suggesting it learns robust features rather than just memorizing the dataset.
Insights
The authors found that predicting a mask from scratch is harder than selecting from a set of learned bases. By forcing the model to generate bases and coefficients separately, visual conditioning becomes much more effective. Furthermore, the audio separation task actually helps the visual segmentation task; to minimize loss, the model is forced to refine its visual attention to the specific object responsible for the sound.
Conclusion
This work demonstrates that audio-visual sound separation can be achieved effectively with weak supervision, eliminating the need for external object detectors. While the model currently requires video-level class labels, it paves the way for future research into fully unsupervised systems. Future work could also explore more complex, non-musical environments where the relationship between visual appearance and sound is less distinct, such as crowded urban scenes.
Summary updated