Abstract
We propose DAVIS, a Diffusion model-based Audio-VIusal Separation framework that solves the audio-visual sound source separation task through a generative manner. While existing discriminative methods that perform mask regression have made remarkable progress in this field, they face limitations in capturing the complex data distribution required for high-quality separation of sounds from diverse categories. In contrast, DAVIS leverages a generative diffusion model and a Separation U-Net to synthesize separated magnitudes starting from Gaussian noises, conditioned on both the audio mixture and the visual footage. With its generative objective, DAVIS is better suited to achieving the goal of high-quality sound separation across diverse categories. We compare DAVIS to existing state-of-the-art discriminative audio-visual separation methods on the domain-specific MUSIC dataset and the open-domain AVE dataset, and results show that DAVIS outperforms other methods in separation quality, demonstrating the advantages of our framework for tackling the audio-visual source separation task.
Introduction
Visually-guided sound source separation aims to isolate a specific sound from a noisy mixture using visual cues—for example, picking out the sound of a "barking dog" from a busy street scene. Traditionally, researchers have treated this as a mask-based regression problem: the model tries to predict a digital mask to overlay on the audio spectrogram to filter out unwanted noise. However, this approach struggles with complex data distributions where sounds overlap heavily or where the target sound is significantly suppressed. Regression models often "blur" results or fail to capture the intricate time-frequency structures of diverse sounds like church bells or specific musical instruments.
Methodology
DAVIS shifts the paradigm from regression to generative learning. Instead of calculating a mask, DAVIS uses a diffusion process to synthesize the separated magnitude spectrogram directly. The model iteratively "grows" high-quality separated audio from noise, conditioned on both the audio mixture and the visual footage.
Key technical innovations in the DAVIS framework include:
- Separation U-Net: A specialized architecture designed to handle the unique characteristics of audio spectrograms.
- Convolution-Attention (CA) Blocks: These blocks combine ResNet layers (to capture local patterns) with Attention mechanisms (to capture long-range dependencies), allowing the model to recognize repeating audio patterns at distant time intervals.
- Audio-Visual Feature Interaction Module (FIM): Rather than simple concatenation, this module uses ResNet and Attention blocks to foster deeper interaction between the audio and visual modalities at the bottleneck.
- Silence Mask-Guided Sampling: To prevent the model from "hallucinating" sounds in silent frames, the authors use a threshold-based mask from the original mixture to guide the sampling process, ensuring that silence in the mixture remains silence in the output.
- $L_1$ Denoising Loss: The authors found $L_1$ loss to be more robust than standard $L_2$ loss for the skewed distribution of audio data.
Key Results
The model was evaluated on two primary benchmarks: AVE (Audio-Visual Event), containing 28 diverse real-world categories, and MUSIC, featuring 11 musical instrument categories. DAVIS sets a new bar for audio-visual separation quality across both quantitative and qualitative metrics.
On the MUSIC dataset, DAVIS achieved a Signal-to-Distortion Ratio (SDR) of 11.61 and a Signal-to-Interference Ratio (SIR) of 18.36, significantly outperforming the previous state-of-the-art, iQuery (11.17 SDR / 15.84 SIR). Visualizations demonstrate that DAVIS successfully recovers suppressed time-frequency structures where mask-regression methods (like CCoL and iQuery) completely fail. Furthermore, in a subjective human evaluation, DAVIS's results were preferred 82.5% of the time over strong baselines.
Discussion and Future Work
DAVIS demonstrates that treating separation as an iterative refinement process allows for the recovery of sounds almost entirely buried in the original mixture. Additionally, because DAVIS learns a robust conditional distribution, it can perform zero-shot text-to-audio separation by replacing visual embeddings with CLIP-based text embeddings.
Limitations remain in the AVE dataset, where off-screen sounds cannot be accounted for by visual frames. Future work will explore more complex multi-source scenes and optimize diffusion sampling speed, which is currently slower than one-shot regression methods.
Summary updated