Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows

Shentong Mo, Yibing Song

Introduction

Foley-Flow is a novel framework designed to generate high-fidelity audio that is perfectly synchronized with video content. By leveraging a "masked alignment" strategy to capture fine-grained temporal details and replacing traditional diffusion processes with a dynamic, flow-based generative model, Foley-Flow achieves state-of-the-art performance in both sound quality and rhythmic accuracy while significantly improving inference speed.

The Challenge: Semantics and Rhythmics

Generating "Foley" (sound effects) for silent video requires solving two distinct problems simultaneously: semantic matching and rhythmic synchronization. For instance, a video of a horse requires not just any horse sound, but a "clip-clop" that aligns precisely with the visual impact of hooves on the ground.

Existing approaches often struggle with two major gaps:

  1. The Timing Gap: Standard contrastive learning (like CLIP) matches global concepts (e.g., "dog" to "bark") but often misses the millisecond-level timing needed for perfect sync.
  2. The Efficiency Gap: High-quality generative models typically rely on Diffusion Models, which require iterative denoising steps, making them computationally expensive and slow for real-time applications.

Methodology: The Foley-Flow Framework

Foley-Flow addresses these challenges through two core innovations: Video-Audio Masking Alignment (VAMA) and Generalized Video-Audio Flow (GVAF).

The Foley-Flow framework: VAMA for alignment and GVAF for generation.

Video-Audio Masking Alignment (VAMA)

To bridge the timing gap, VAMA moves beyond global embeddings to a masked autoencoding approach. By masking segments of the audio and requiring the model to reconstruct them using corresponding temporal video segments, the model is forced to learn the precise visual-audio correlations. This "reconstruction-over-contrast" approach ensures that the encoders capture the physics of motion and its associated sound.

Generalized Video-Audio Flow (GVAF)

For generation, Foley-Flow utilizes a flow-based model (specifically velocity flows) instead of diffusion. GVAF introduces Dynamic Conditioning, where the flow is guided by temporally varying video features rather than a static prompt. This allows the generation process to "track" visual motion as it unfolds. Furthermore, flow-based models are significantly more efficient, often reaching high quality in far fewer steps than diffusion models.

Experimental Results

The model was evaluated on the VGGSound and AudioSet benchmarks. Foley-Flow demonstrated superior performance across all metrics, significantly outperforming previous state-of-the-art models like Diff-Foley, VATT, and FoleyGen.

Quantitative comparison of Foley-Flow against state-of-the-art methods on VGGSound.

Key performance highlights include:

  • Semantic Accuracy (KLD): Achieved a score of 0.97, a substantial improvement over the previous best of 2.25 (VATT). Lower is better, indicating the sound matches the video's concept closely.
  • Audio Quality (FAD): Reached 0.52, nearly halving the error of the previous leader, V2A-Mapper (0.99).
  • Rhythmic Alignment (Align Acc): Achieved a staggering 98.97% accuracy, suggesting that the generated audio is nearly indistinguishable from the ground truth in terms of synchronization.

Conclusion and Future Work

Foley-Flow represents a significant step forward in video-to-audio generation by prioritizing temporal reconstruction and dynamic flow conditioning. While highly effective for 10-second clips, future research may explore long-form generation and the handling of complex, multi-layered audio environments. The efficiency of the flow-based framework also opens doors for real-time applications in gaming and virtual reality.

Summary updated

Abstract

Coordinated audio generation based on video inputs typically requires a strict audio-visual (AV) alignment, where both semantics and rhythmics of the generated audio segments shall correspond to those in the video frames. Previous studies leverage a two-stage design where the AV encoders are firstly aligned via contrastive learning, then the encoded video representations guide the audio generation process. We observe that both contrastive learning and global video guidance are effective in aligning overall AV semantics while limiting temporally rhythmic synchronization. In this work, we propose FoleyFlow to first align unimodal AV encoders via masked modeling training, where the masked audio segments are recovered under the guidance of the corresponding video segments. After training, the AV encoders which are separately pretrained using only unimodal data are aligned with semantic and rhythmic consistency. Then, we develop a dynamic conditional flow for the final audio generation. Built upon the efficient velocity flow generation framework, our dynamic conditional flow utilizes temporally varying video features as the dynamic condition to guide corresponding audio segment generations. To this end, we extract coherent semantic and rhythmic representations during masked AV alignment, and use this representation of video segments to guide audio generation temporally. Our audio results are evaluated on the standard benchmarks and largely surpass existing results under several metrics. The superior performance indicates that FoleyFlow is effective in generating coordinated audios that are both semantically and rhythmically coherent to various video sequences.
BibTeX
@inproceedings{mo2025foley,
  author = {Shentong Mo and Yibing Song},
  title = {{Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02692},
}
Figure