Weakly-Supervised Audio-Visual Sound Source Detection and Separation

Tanzila Rahman, Leonid Sigal

TL;DR

This paper introduces an end-to-end framework for localizing and separating individual sound sources in video using only video-level labels (e.g., "cello," "flute"). By decomposing the audio separation task into a set of learned "mask bases" and dynamically predicted coefficients, the model achieves state-of-the-art performance without requiring expensive bounding box annotations or pre-trained object detectors.

What Problem Does This Paper Address?

Isolating a specific sound from a noisy environment—the "cocktail party effect"—is a natural human ability but a significant challenge for AI. While recent "visually-guided" methods have made strides, they often suffer from two major drawbacks:

  1. Heavy Supervision: Many models rely on pre-trained object detectors or manual bounding box annotations to "tell" the network where to look.
  2. Simplistic Assumptions: Earlier self-supervised methods often assumed a video contains only a single sound source, failing in complex, multi-instrument scenes.

The authors tackle the challenge of weakly-supervised co-segmentation: learning what an object looks like and how it sounds simultaneously, using only the presence/absence labels of objects in a video.

Methods

The architecture consists of two primary paths that interact to perform "Mix-and-Separate" self-supervision (where the model learns to recover original audio from an artificial mixture of two videos).

  • Video Frame Semantic Segmentation Path: Using a ResNet-18 backbone, the model generates class-specific attention maps. It employs a dual-attention strategy: an "Expansive" detector to identify the general object region and a "Discriminative" detector to find the most salient parts. These are combined to create a final attention map ($X_m$) used to pool visual features.
  • Spectrogram Mask Prediction Path: Instead of predicting a final audio mask directly, the model uses an Attention U-Net to generate $k$ "audio mask bases."
  • The "Aha" Component (Mask Coefficient Generator): The model fuses the pooled visual features with the audio features to predict a set of sparse coefficients. The final spectrogram mask is then constructed as a linear combination of the mask bases weighted by these coefficients. This bi-linear decomposition allows the network to learn a general "vocabulary" of sounds (bases) and use visual cues to "dial in" the right combination for a specific object.

Datasets

  • MUSIC Dataset: A collection of 685 untrimmed videos of musical solos and duets (e.g., accordion, cello, xylophone).
  • AudioSet-SingleSource: A subset of the massive AudioSet, used primarily for cross-dataset evaluation to test the model's ability to generalize to unseen instruments and environments.

Key Insights and Ideas

  • Decomposition is Key: The most significant insight is that predicting a mask from scratch is harder than selecting from a set of learned bases. By forcing the model to generate bases and coefficients separately, the authors found the visual conditioning becomes much more effective.
  • Mutual Regularization: The audio separation task actually helps the visual segmentation task. Because the model must separate the sound correctly to minimize loss, it is forced to refine its visual attention to the specific object responsible for that sound.
  • Eliminating External Dependencies: Unlike prior state-of-the-art methods (like Gao et al.'s Co-Separation), this model does not need an external object detector trained on ImageNet or COCO. It "discovers" the objects through the synergy of audio and video.

Key Results

The authors evaluated their model using standard metrics: Signal-to-Distortion Ratio (SDR), Signal-to-Interference Ratio (SIR), and Signal-to-Artifact Ratio (SAR).

  • State-of-the-Art Performance: On the MUSIC dataset, the proposed method achieved an SDR of 9.14 (using ratio masks), significantly outperforming the "Sound-of-Pixels" baseline (7.81) and "CO-SEPARATION" (7.64).
  • High Localization Accuracy: Despite only having video-level labels, the model achieved up to 93.69% classification accuracy, proving that its internal attention maps were successfully localizing the correct instruments.
  • Generalization: In cross-dataset tests (training on MUSIC, testing on AudioSet), the model showed a nearly 10x performance increase in SDR compared to previous baselines, suggesting it learns robust features rather than just memorizing the dataset.

Limitations and Future Work

While the model removes the need for bounding boxes, it still requires video-level class labels during training. A truly "in-the-wild" system would ideally function with zero labels (fully unsupervised).

Additionally, the paper focuses primarily on musical instruments. Future work could explore more complex, non-musical environments where the relationship between visual appearance and sound is less distinct (e.g., a crowded street or a kitchen). The authors also note that while the model handles multiple sources, the "Mix-and-Separate" paradigm still relies on the synthetic mixing of clean tracks, which may not perfectly mirror the complexity of real-world acoustic overlaps.

Summary updated

Abstract

Learning how to localize and separate individual object sounds in the audio channel of the video is a difficult task. Current state-of-the-art methods predict audio masks from artificially mixed spectrograms, known as Mix-and-Separate framework. We propose an audio-visual co-segmentation, where the network learns both what individual objects look and sound like, from videos labeled with only object labels. Unlike other recent visually-guided audio source separation frameworks, our architecture can be learned in an end-to-end manner and requires no additional supervision or bounding box proposals. Specifically, we introduce weakly-supervised object segmentation in the context of sound separation. We also formulate spectrogram mask prediction using a set of learned mask bases, which combine using coefficients conditioned on the output of object segmentation — a design that facilitates separation. Extensive experiments on the MUSIC dataset show that our proposed approach outperforms state-of-the-art methods on visually guided sound source separation and sound denoising.
BibTeX
@inproceedings{rahman2021weakly,
  author = {Tanzila Rahman and Leonid Sigal},
  title = {{Weakly-Supervised Audio-Visual Sound Source Detection and Separation}},
  booktitle = {ICME},
  year = {2021},
  url = {https://openreview.net/forum?id=kuk6lD-PEm},
}

Loading related content...

Figure