Abstract
Visual acoustic matching (VAM) and dereverberation are critical for immersive AR/VR and speech intelligibility. However, data scarcity and the limitations of GAN-based models hinder progress. We present MVSD, a framework that treats these tasks as reciprocal operations within a mutual learning loop. By employing visual scene-driven diffusion models, MVSD captures complex acoustic reflections and learns effectively from unpaired data, achieving state-of-the-art performance in both audio realism and speech clarity.
Introduction
Creating immersive audio environments requires matching "dry" (anechoic) audio to specific visual scenes—a process known as Visual Acoustic Matching (VAM). Conversely, dereverberation is necessary to remove echoes for improved speech intelligibility. Current research faces two primary challenges: the difficulty of collecting paired anechoic and reverberant data in real-world settings, and the tendency of GAN-based models to produce over-smoothed audio that lacks realistic acoustic detail.
Methodology
The MVSD framework addresses these challenges through two core innovations: a mutual learning loop and a visual scene-driven diffusion process.
Mutual Learning Loop
MVSD treats VAM and dereverberation as inverse operations. A "reverberator" adds scene-specific echoes to anechoic audio, while a "dereverberator" removes them. By forming a closed loop (Anechoic $\rightarrow$ Reverb $\rightarrow$ Anechoic), the models act as mutual evaluators. This reciprocity allows the system to exploit "unpaired" data—audio that is either exclusively dry or reverberant—greatly expanding the available training signal.
Visual Scene-driven Diffusion (VSD)
To capture the stochastic and complex nature of acoustic reflections, MVSD utilizes diffusion models instead of GANs. These models generate audio by iteratively denoising a signal, guided by visual embeddings from a ResNet-18 encoder. A controllable UNet architecture with cross-modal attention ensures that the generated audio features are precisely aligned with environmental cues like room size and surface materials.
Experiments and Results
The model was evaluated on the SoundSpaces-Speech (simulated) and Acoustic AVSpeech (real-world) datasets, supplemented by anechoic audio from LibriSpeech to further boost performance.
Performance Metrics
MVSD achieved state-of-the-art results across multiple benchmarks:
- Acoustic Accuracy: Reduced STFT-distance by 23.6% on the SoundSpaces-Speech test set compared to previous methods.
- Speech Intelligibility: Lowered the Word Error Rate (WER) in dereverberation tasks to 4.27%, outperforming the previous best of 4.44%.
- Unpaired Data Utility: Incorporating unpaired data led to a 9.1% relative improvement in matching room acoustics (RTE).
- Human Evaluation: User studies confirmed a strong preference for MVSD, with participants noting superior naturalness and visual-acoustic alignment.
Conclusion and Future Work
MVSD demonstrates that reciprocity and diffusion are powerful tools for visual acoustics. While the model sets new performance standards, its computational complexity leads to an inference time of approximately 1.09 seconds per clip, which is currently a barrier for real-time applications. Future research will focus on model distillation to enhance speed and reduce the parameter footprint for deployment on edge devices.
Summary updated