Introduction
Foley-Flow is a novel framework designed to generate high-fidelity audio that is perfectly synchronized with video content. By leveraging a "masked alignment" strategy to capture fine-grained temporal details and replacing traditional diffusion processes with a dynamic, flow-based generative model, Foley-Flow achieves state-of-the-art performance in both sound quality and rhythmic accuracy while significantly improving inference speed.
The Challenge: Semantics and Rhythmics
Generating "Foley" (sound effects) for silent video requires solving two distinct problems simultaneously: semantic matching and rhythmic synchronization. For instance, a video of a horse requires not just any horse sound, but a "clip-clop" that aligns precisely with the visual impact of hooves on the ground.
Existing approaches often struggle with two major gaps:
- The Timing Gap: Standard contrastive learning (like CLIP) matches global concepts (e.g., "dog" to "bark") but often misses the millisecond-level timing needed for perfect sync.
- The Efficiency Gap: High-quality generative models typically rely on Diffusion Models, which require iterative denoising steps, making them computationally expensive and slow for real-time applications.
Methodology: The Foley-Flow Framework
Foley-Flow addresses these challenges through two core innovations: Video-Audio Masking Alignment (VAMA) and Generalized Video-Audio Flow (GVAF).
Video-Audio Masking Alignment (VAMA)
To bridge the timing gap, VAMA moves beyond global embeddings to a masked autoencoding approach. By masking segments of the audio and requiring the model to reconstruct them using corresponding temporal video segments, the model is forced to learn the precise visual-audio correlations. This "reconstruction-over-contrast" approach ensures that the encoders capture the physics of motion and its associated sound.
Generalized Video-Audio Flow (GVAF)
For generation, Foley-Flow utilizes a flow-based model (specifically velocity flows) instead of diffusion. GVAF introduces Dynamic Conditioning, where the flow is guided by temporally varying video features rather than a static prompt. This allows the generation process to "track" visual motion as it unfolds. Furthermore, flow-based models are significantly more efficient, often reaching high quality in far fewer steps than diffusion models.
Experimental Results
The model was evaluated on the VGGSound and AudioSet benchmarks. Foley-Flow demonstrated superior performance across all metrics, significantly outperforming previous state-of-the-art models like Diff-Foley, VATT, and FoleyGen.
Key performance highlights include:
- Semantic Accuracy (KLD): Achieved a score of 0.97, a substantial improvement over the previous best of 2.25 (VATT). Lower is better, indicating the sound matches the video's concept closely.
- Audio Quality (FAD): Reached 0.52, nearly halving the error of the previous leader, V2A-Mapper (0.99).
- Rhythmic Alignment (Align Acc): Achieved a staggering 98.97% accuracy, suggesting that the generated audio is nearly indistinguishable from the ground truth in terms of synchronization.
Conclusion and Future Work
Foley-Flow represents a significant step forward in video-to-audio generation by prioritizing temporal reconstruction and dynamic flow conditioning. While highly effective for 10-second clips, future research may explore long-form generation and the handling of complex, multi-layered audio environments. The efficiency of the flow-based framework also opens doors for real-time applications in gaming and virtual reality.
Summary updated