Mutual Learning for Acoustic Matching and Dereverberation via Visual Scene-driven Diffusion

Jian Ma, Wenguan Wang, Yi Yang, Feng Zheng

Abstract

Visual acoustic matching (VAM) and dereverberation are critical for immersive AR/VR and speech intelligibility. However, data scarcity and the limitations of GAN-based models hinder progress. We present MVSD, a framework that treats these tasks as reciprocal operations within a mutual learning loop. By employing visual scene-driven diffusion models, MVSD captures complex acoustic reflections and learns effectively from unpaired data, achieving state-of-the-art performance in both audio realism and speech clarity.

Introduction

Creating immersive audio environments requires matching "dry" (anechoic) audio to specific visual scenes—a process known as Visual Acoustic Matching (VAM). Conversely, dereverberation is necessary to remove echoes for improved speech intelligibility. Current research faces two primary challenges: the difficulty of collecting paired anechoic and reverberant data in real-world settings, and the tendency of GAN-based models to produce over-smoothed audio that lacks realistic acoustic detail.

Methodology

The MVSD framework addresses these challenges through two core innovations: a mutual learning loop and a visual scene-driven diffusion process.

Mutual Learning Loop

MVSD treats VAM and dereverberation as inverse operations. A "reverberator" adds scene-specific echoes to anechoic audio, while a "dereverberator" removes them. By forming a closed loop (Anechoic $\rightarrow$ Reverb $\rightarrow$ Anechoic), the models act as mutual evaluators. This reciprocity allows the system to exploit "unpaired" data—audio that is either exclusively dry or reverberant—greatly expanding the available training signal.

The MVSD framework treats reverberation and dereverberation as reciprocal tasks, enabling training on both paired and unpaired audio data.

Visual Scene-driven Diffusion (VSD)

To capture the stochastic and complex nature of acoustic reflections, MVSD utilizes diffusion models instead of GANs. These models generate audio by iteratively denoising a signal, guided by visual embeddings from a ResNet-18 encoder. A controllable UNet architecture with cross-modal attention ensures that the generated audio features are precisely aligned with environmental cues like room size and surface materials.

Experiments and Results

The model was evaluated on the SoundSpaces-Speech (simulated) and Acoustic AVSpeech (real-world) datasets, supplemented by anechoic audio from LibriSpeech to further boost performance.

Performance Metrics

MVSD achieved state-of-the-art results across multiple benchmarks:

  • Acoustic Accuracy: Reduced STFT-distance by 23.6% on the SoundSpaces-Speech test set compared to previous methods.
  • Speech Intelligibility: Lowered the Word Error Rate (WER) in dereverberation tasks to 4.27%, outperforming the previous best of 4.44%.
  • Unpaired Data Utility: Incorporating unpaired data led to a 9.1% relative improvement in matching room acoustics (RTE).
  • Human Evaluation: User studies confirmed a strong preference for MVSD, with participants noting superior naturalness and visual-acoustic alignment.

Conclusion and Future Work

MVSD demonstrates that reciprocity and diffusion are powerful tools for visual acoustics. While the model sets new performance standards, its computational complexity leads to an inference time of approximately 1.09 seconds per clip, which is currently a barrier for real-time applications. Future research will focus on model distillation to enhance speed and reduce the parameter footprint for deployment on edge devices.

Summary updated

BibTeX
@inproceedings{ma2024mutual,
  author = {Jian Ma and Wenguan Wang and Yi Yang and Feng Zheng},
  title = {{Mutual Learning for Acoustic Matching and Dereverberation via Visual Scene-driven Diffusion}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72649-1\_19},
}
Figure