Abstract
Generating realistic audio for silent videos—Video-to-Audio (V2A) generation—is a challenging task that traditionally requires computationally expensive joint training on massive paired datasets. This paper introduces Multimodal Diffusion Guidance (MDG), a training-free method that steers pretrained audio diffusion models using a novel geometric objective. By minimizing the volume of the parallelotope spanned by video, audio, and text embeddings in a shared latent space, MDG achieves superior semantic alignment and audio quality. Evaluations on the VGGSound and AudioCaps benchmarks show that MDG significantly outperforms existing training-free baselines in both in-domain and out-of-domain scenarios.
Introduction
The ability to automatically generate high-fidelity audio for silent video has transformative potential for industries ranging from automated video editing and Foley sound design to accessibility tools for the visually impaired. However, the core challenge of V2A generation lies in achieving precise synchronization and semantic coherence between visual events and their corresponding sounds.
Historically, state-of-the-art performance has been dominated by models trained from scratch on large-scale multimodal datasets. While effective, this approach is resource-intensive and lacks flexibility. Recent "training-free" alternatives attempt to repurpose pretrained models by using pairwise similarity (e.g., cosine similarity between video and audio) to guide the generation process. However, these methods often fail to maintain global coherence across all modalities simultaneously, frequently resulting in "semantic drift" where the generated audio becomes generic noise.
Multimodal Diffusion Guidance (MDG)
To address the limitations of pairwise guidance, the authors propose Multimodal Diffusion Guidance (MDG). MDG is a plug-and-play control signal designed to work with pretrained latent diffusion models, such as AudioLDM.
The fundamental innovation of MDG is the use of a volume-based tri-modal alignment objective. Instead of checking if audio matches video and then separately checking if audio matches text, MDG considers the joint relationship between all three. Using pretrained encoders, the video frames, text prompt, and the current audio latent are mapped to a shared embedding space. These three embeddings are treated as vectors, and MDG calculates the volume of the geometric shape (a parallelotope) they span.
Mathematically, if the video, text, and audio represent the same semantic concept, their vectors will be highly correlated and point in a similar direction, causing the spanned volume to shrink toward zero. During the denoising loop of the diffusion model, MDG calculates the gradient of this volume and updates the audio latent to minimize it. This forces the generation to converge on a representation that is jointly compatible with both the visual and textual context without any retraining of the underlying model.
Experimental Evaluation
The effectiveness of MDG was evaluated on two primary benchmarks:
- VGGSound: A large-scale dataset of video clips with specific sound events, used for in-domain testing (3,000 samples).
- AudioCaps: A more challenging dataset where the sound source might not be visually present, used to test out-of-domain generalization (697 samples).
The performance was measured using standard metrics, including Fréchet Audio Distance (FAD) for audio quality, Fréchet Audio-Visual Distance (FAVD) for coherence, and PEAVS for perceptual alignment.
Results and Discussion
MDG demonstrated substantial improvements over existing baselines, including Seeing&Hearing, SpecVQGAN, and Diff-Foley. On the VGGSound dataset, MDG achieved an FAD of 6.04, significantly better than the 7.80 achieved by the pairwise-guided Seeing&Hearing model. Audio-visual coherence also saw marked improvements, with FAVD dropping from 3.44 to 2.60.
Qualitative analysis via spectrograms reveals that MDG produces distinct, structured soundscapes. For instance, in underwater scenes, MDG generates clear, rhythmic bubbling sounds, whereas baseline methods often collapse into undifferentiated broadband noise. This highlights the power of volume-based guidance in preventing semantic drift.
The results suggest two key insights:
- Geometric Volume is Superior: Enforcing joint consistency through volume minimization is more effective than independent pairwise constraints. It ensures that all modalities exist in the same semantic neighborhood.
- Modularity and Efficiency: High-quality multimodal grounding can be achieved through lightweight mathematical constraints during inference, bypassing the need for expensive retraining.
Conclusion
Multimodal Diffusion Guidance provides a robust, training-free framework for high-fidelity Video-to-Audio generation. By shifting from pairwise similarity to a joint volume-based alignment, the method achieves state-of-the-art performance in semantic grounding and audio quality. While the method's performance is tied to the quality of the underlying pretrained encoders, its modularity offers a promising path for future research in cross-modal generation and multi-sensory AI systems.
Summary updated