DAVIS: High-Quality Audio-Visual Separation with Generative Diffusion Models

Chao Huang, Susan Liang, Yapeng Tian, Anurag Kumar, Chenliang XuSubmission to ICLR 2024 · Withdrawn

TL;DR

DAVIS is the first framework to reframe visually-guided sound source separation as a conditional generation problem solved via diffusion models. Rather than predicting separation masks through discriminative regression, DAVIS generates clean magnitude spectrograms directly from Gaussian noise conditioned on video frames and the mixed audio. This generative formulation significantly improves separation quality, boosting Signal-to-Interference Ratio (SIR) by up to 3.1 dB over prior state-of-the-art approaches on open- and closed-domain benchmarks.

What Problem Does This Paper Address?

Audio-visual source separation aims to isolate individual acoustic streams from a complex audio mixture using synchronized video (e.g., isolating a guitar track from a band performance by watching the guitarist).

Traditional approaches treat this discriminatively—training a neural network to predict a time-frequency mask that filters the input mixture. While effective for clean, domain-specific audio (such as solo musical instruments), mask regression struggles with open-domain audio filled with background noise, off-screen speakers, and overlapping acoustic textures. These complex time-frequency distributions are difficult to model via direct regression. DAVIS asks: Can we achieve higher-fidelity separation by treating the problem as conditional generative modeling instead?

Methods

DAVIS models source separation as a $T$-step conditional reverse diffusion process in the STFT magnitude spectrogram domain:

Gaussian Noise ───► [ Separation U-Net (CA Blocks) ] ───► Clean Separated Spectrogram
                          ▲                ▲
                    Audio Mixture    Visual Features
                                     (ResNet + Transformer + FIM)
  1. Spectrogram-Domain Diffusion: During training, noise is added to the ground-truth target spectrogram via a non-linear schedule. At inference, the network starts from pure Gaussian noise and iteratively denoises it over 25 DDIM steps into the target source spectrogram.
  2. Separation U-Net with CA Blocks: Standard convolutions miss long-range acoustic relationships. DAVIS incorporates Convolution-Attention (CA) blocks, which combine:
  3. A ResNet block (weight-standardized 2D convolution) for local time-frequency structures.
  4. An efficient Time-Frequency Attention block for long-range cross-frequency patterns.
  5. A Time Attention block to track temporal evolution.
  6. Temporal Visual Transformer: Video frames are passed through a frozen ResNet-18 visual encoder and aggregated using a shallow temporal transformer to filter out unaligned or visually irrelevant frames.
  7. Feature Interaction Module (FIM): Rather than naively concatenating audio and visual embeddings at the U-Net bottleneck, the FIM tiles visual embeddings across spatial dimensions and processes them with joint ResNet and attention blocks to enforce local and global cross-modal alignment.
  8. Non-Linear Noise Scheduling: Standard linear variance schedules fail because spectrograms are sparse with concentrated energy pockets. DAVIS adopts non-linear (sigmoid/cosine) noise schedules to prevent early noise from drowning out the audio signal.

Datasets

The authors evaluate DAVIS on two standard benchmarks without creating new datasets:

  • MUSIC Dataset: Domain-specific benchmark covering 11 musical instrument classes in clean solo performances (468 train / 26 val / 26 test videos).
  • Audio-Visual Event (AVE) Dataset: Open-domain, unconstrained benchmark containing 28 diverse event classes (e.g., Barking, Church Bell, Frying food) across 4,143 10-second clips (3,339 train / 402 val / 402 test). AVE contains real-world complications like off-screen human speech and ambient noise.

Key Insights and Ideas

  • Generative vs. Discriminative Separation: Predicting spectrogram magnitudes from noise—conditioned on the mixture—allows the model to navigate multi-modal acoustic distributions far better than regressing an explicit ratio mask.
  • Spectrogram Sparsity Requires Non-Linear Noise Schedules: Conventional linear noise schedules completely failed to train (resulting in negative SDR). Because magnitude spectrograms are sparse, non-linear schedules (sigmoid/cosine) are mandatory to preserve signal clues across intermediate diffusion steps.
  • Cross-Modal Attention Over Pure Bottleneck Concatenation: Merely concatenating visual vectors to the bottleneck features yielded an SDR of 10.85 dB; adding dedicated feature interaction (FIM) pushed performance to 11.88 dB on MUSIC validation.
  • Separation Beyond Binary Mixtures: Even though trained strictly on two-source artificial mixes ($a^{(1)} + a^{(2)}$), DAVIS successfully separates target sounds from natural mixtures containing three or more simultaneous audio sources (e.g., isolating a target sound while ignoring concurrent off-screen speech and traffic).

Key Results

DAVIS was evaluated using standard metrics: Signal-to-Distortion Ratio (SDR) and Signal-to-Interference Ratio (SIR).

  • MUSIC: DAVIS achieved 11.18 dB SDR and 18.06 dB SIR, outperforming all prior methods. It beat the previous state-of-the-art method (iQuery, 14.95 dB SIR) by +3.11 dB SIR, notably doing so without requiring class labels during training or inference.
  • AVE (Open-Domain): DAVIS achieved 3.77 dB SDR and 7.95 dB SIR, outperforming iQuery (6.82 dB SIR) by +1.13 dB SIR and doubling the SIR of earlier systems like CCoL (3.25 dB) and SoP (4.17 dB).
  • Inference Efficiency: Thanks to DDIM sampling, DAVIS produces high-fidelity audio in 25 steps (with competitive separation already visible at 10 steps: 11.82 dB SDR), mitigating diffusion's typical sampling latency bottleneck.

Limitations and Future Work

  • Global Visual Embedding Reliance: DAVIS condenses video frames into a single global visual vector. While sufficient for solo videos or single-event scenes, it struggles when multiple distinct sounding objects appear simultaneously on-screen.
  • Lack of Explicit Sound Localization: The framework does not explicitly locate where the sound originates in the pixel space. The authors highlight joint audio-visual localization and separation within a unified generative diffusion architecture as an important future direction.

Summary updated

Abstract

We propose DAVIS, a Diffusion model-based Audio-VIusal Separation framework that solves the audio-visual sound source separation task through a generative manner. While existing discriminative methods that perform mask regression have made remarkable progress in this field, they face limitations in capturing the complex data distribution required for high-quality separation of sounds from diverse categories. In contrast, DAVIS leverages a generative diffusion model and a Separation U-Net to synthesize separated magnitudes starting from Gaussian noises, conditioned on both the audio mixture and the visual footage. With its generative objective, DAVIS is better suited to achieving the goal of high-quality sound separation across diverse categories. We compare DAVIS to existing state-of-the-art discriminative audio-visual separation methods on the domain-specific MUSIC dataset and the open-domain AVE dataset, and results show that DAVIS outperforms other methods in separation quality, demonstrating the advantages of our framework for tackling the audio-visual source separation task.
BibTeX
@misc{huang2023davis,
  author = {Chao Huang and Susan Liang and Yapeng Tian and Anurag Kumar and Chenliang Xu},
  title = {{DAVIS: High-Quality Audio-Visual Separation with Generative Diffusion Models}},
  year = {2023},
  eprint = {2308.00122},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.00122},
}

Loading related content...

Figure