V-SlowFast Network for Efficient Visual Sound Separation

Lingyu Zhu, Esa RahtuWACV 2022

TL;DR

This paper introduces V-SlowFast, a novel three-stream network that leverages visual information to separate sounds from audio mixtures. By processing visual frames alongside both slow and fast temporal resolution spectrograms, V-SlowFast achieves state-of-the-art performance while significantly reducing model complexity and computational cost.

What Problem Does This Paper Address?

The task of sound source separation, aiming to isolate individual sounds from a mixed audio signal, is a long-standing challenge in audio processing. While traditional methods rely solely on audio cues, recent research has explored incorporating visual information (e.g., a talking face or a playing instrument) to aid this task. However, existing visual sound separation methods often rely on full-resolution spectrograms, leading to computationally expensive and complex models. This paper addresses the need for more efficient and effective visual sound separation by proposing a novel network architecture that effectively utilizes visual cues across different temporal resolutions of audio spectrograms.

Methods

The core of the V-SlowFast network is its three-stream architecture:

  • Vision Stream: Processes a single visual frame from the input video. This stream is enhanced with two novel contrastive learning objectives: an embedding objective to learn discriminative visual features and a localization objective to predict the presence of a sound source.
  • Slow Spectrogram Stream: Operates on a low temporal resolution spectrogram of the audio mixture. This stream captures coarse temporal information.
  • Fast Spectrogram Stream: Works with a high temporal resolution spectrogram, capturing fine-grained temporal details. This stream acts as a residual network, refining the separation from the slow stream.

A key innovation is the Audio-Visual Global Attention (AVGA) module, which fuses visual embeddings with sound features. This module allows the network to focus on relevant visual cues for the target sound source. The network is trained using a self-supervised approach, combining a contrastive learning objective for the vision stream with a sound separation objective for the entire system.

Datasets

The paper evaluates V-SlowFast on the following datasets:

  • MUSIC-21: A dataset containing videos of musical instruments.
  • AVE (Audio-Visual Events): A subset of AudioSet focusing on audio-visual event categories.
  • VGG-Sound: A large-scale dataset with diverse audio-visual categories.

The authors also mention collecting 100 natural background audios from YouTube to create mixtures for evaluating separation from background noise.

Key Insights and Ideas

  • Temporal Resolution Matters: Processing audio spectrograms at both coarse (slow) and fine (fast) temporal resolutions is crucial for effective sound separation, offering complementary information.
  • Visual Cues Enhance Efficiency: Incorporating visual information not only improves separation performance but also allows for significantly more compact and computationally efficient models compared to purely audio-based or previous visual methods.
  • Contrastive Learning for Vision: Applying contrastive learning to the visual stream helps learn more discriminative features, which are vital for identifying and localizing sound sources.
  • Global Attention for Fusion: The AVGA module effectively bridges the gap between visual and audio modalities, enabling the network to leverage relevant visual context.

Key Results

  • State-of-the-Art Performance: V-SlowFast outperforms previous state-of-the-art methods on MUSIC-21, AVE, and VGG-Sound datasets, achieving higher SDR (Signal-to-Distortion Ratio) scores. For example, V-SlowFast (1) achieves 10.89 dB SDR on MUSIC-21, surpassing the previous best of 9.50 dB.
  • Significant Efficiency Gains: A smaller V-SlowFast architecture variant (V-SlowFast (2)) achieves comparable results to multi-stage models but with a 74.2% reduction in model parameters and an 81.4% reduction in GMACs (Giga Multiply-Accumulate Operations).
  • Ablation Studies: The paper demonstrates that using both slow and fast spectrogram streams yields better results than using either stream alone. It also shows that the choice of temporal resolution (α) for the spectrograms impacts performance, with moderate downsampling rates generally performing best.

Limitations and Future Work

  • Dataset Dependency: While V-SlowFast performs well on benchmark datasets, its performance in highly complex or unseen natural scenarios might still be limited, especially when ground truth for sources is unavailable.
  • Computational Cost of Fast Stream: The paper notes that the performance of the fast spectrogram network significantly influences the overall result, implying that further optimization of this stream could lead to even greater improvements.
  • Future Work: The authors suggest exploring deeper U-Net architectures for the sound network and investigating the "V-FastSlow" approach (where fast spectrograms are processed first) for further performance gains. They also highlight the potential for using more efficient vision backbones like MV2.

Summary updated

BibTeX
@inproceedings{zhu2022v,
  author = {Lingyu Zhu and Esa Rahtu},
  title = {{V-SlowFast Network for Efficient Visual Sound Separation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure