TL;DR
This paper introduces Smooth-Foley, a novel approach to generating continuous, synchronized Foley sounds for videos. By leveraging semantic guidance from textual labels and integrating fine-grained frame-wise video features, Smooth-Foley significantly improves temporal and semantic alignment compared to existing methods, particularly for complex continuous sound scenarios.
What Problem Does This Paper Address?
Creating realistic and synchronized audio for videos, especially Foley sounds (everyday sounds like footsteps, doors opening, etc.), is crucial for immersive content but is a labor-intensive process. While recent video-to-audio (V2A) generation models have made strides, they struggle with two key challenges:
- Temporal Alignment: Ensuring generated sounds precisely match the timing of events in the video. Existing methods often have inaccurate temporal conditions, leading to sounds that are slightly off or miss critical moments.
- Semantic Alignment & Continuous Sounds: Accurately identifying and generating sounds that correspond to visual cues, especially for continuous sounds (like a siren or an airplane's engine) or when objects become visually ambiguous (e.g., a distant airplane). Current models often fail to maintain consistent sound generation for these scenarios.
The paper aims to bridge this gap by enhancing both the semantic and temporal accuracy of V2A generation, enabling the creation of smoother and more contextually relevant audio for videos.
Methods
Smooth-Foley builds upon existing pre-trained text-to-audio (T2A) generation models, adapting them for V2A tasks using lightweight adapters. The core innovations lie in how it incorporates visual and textual information:
- Semantic Guidance: This is a two-pronged approach:
- Frame-wise Visual Guidance: Instead of using aggregated, low-resolution video clips (clip-wise), Smooth-Foley processes individual video frames. This provides a much higher temporal resolution of visual information, allowing for more precise understanding of subtle visual changes that correspond to sound events.
- Textual Label Guidance: The textual label associated with the video (e.g., "train horn") is used to refine temporal conditions. By comparing frame embeddings with the textual label using CLIP, the model can better pinpoint the occurrence of specific sound-generating objects, even when visual cues are weak.
- Frame Adapter: This module integrates the high-resolution, frame-wise visual features into the T2A model. It uses a learnable projection layer to adapt features from a pre-trained CLIP image encoder and then employs parallel cross-attention to fuse these frame embeddings with textual embeddings.
- Temporal Adapter: This module enhances temporal alignment by leveraging the textual label. It computes cosine similarities between projected frame embeddings and the textual embedding. These similarities, binarized with a threshold, provide a more accurate temporal condition for the audio generator. This adapter shares architecture with UNet encoders used in ControlNet.
Both adapters are trained separately, allowing for efficient fine-tuning of pre-trained T2A models.
Datasets
- VGGSound-Continuous: The authors introduce a new filtered subset of the VGGSound dataset. This subset specifically targets video clips that contain continuous sounds (e.g., sirens, airplane sounds) or objects that produce such sounds. This dataset was created by selecting clips with relevant labels and then filtering them to ensure the audio content accurately matched the labels, using text-to-audio grounding techniques. They manually curated 95 challenging clips for their test split.
- VGGSound: The full VGGSound dataset is also used for evaluation.
Key Insights and Ideas
- Frame-wise is Better than Clip-wise for Temporal Accuracy: The paper strongly suggests that processing video at a frame-by-frame level, rather than in larger clips, is critical for achieving precise temporal synchronization in V2A generation. This higher granularity captures subtle visual cues that are missed by lower-resolution approaches.
- Textual Labels as a Temporal Anchor: Using textual labels not just for semantic understanding but also to guide temporal alignment is a powerful idea. It acts as an explicit signal to help the model focus on the correct moments, especially when visual information is ambiguous or when multiple sound events are present.
- Semantic Guidance Enhances Physical Realism: The improved temporal and semantic alignment leads to audio that adheres better to physical laws, such as the Doppler effect for moving objects, as demonstrated in their qualitative examples.
- Targeted Datasets Improve Performance: Creating a specialized dataset like VGGSound-Continuous, focused on challenging continuous sound scenarios, allows for more effective fine-tuning and evaluation of models designed for these specific tasks.
Key Results
- Superior Performance: Smooth-Foley consistently outperforms baseline models like FoleyCrafter and Diff-Foley across various metrics.
- Quantitative Gains:
- On the VGGSound-Continuous dataset, Smooth-Foley achieved the lowest Mean KL Divergence (MKL) and highest CLIP Score, indicating better semantic alignment and audio-video consistency. For example, MKL was 1.558 (vs. 1.855 for FC w/o prompt) and CLIP Score was 55.124 (vs. 54.400 for FC w/ prompt).
- Frechet Audio Distance (FAD) results also showed significant improvements, with Smooth-Foley achieving much lower FAD scores (e.g., 0.18/0.11 vs. 0.25/0.14 for FC w. prompt).
- Qualitative Improvements: In qualitative comparisons, Smooth-Foley successfully generated sounds for challenging scenarios where other models failed, such as generating engine noise for an approaching airplane when it was visually blurred, or accurately capturing continuous siren sounds. It also demonstrated adherence to physical principles like the Doppler effect.
- Subjective Evaluation: Human evaluators rated Smooth-Foley significantly higher in semantic alignment, temporal alignment, and audio quality compared to baselines.
Limitations and Future Work
- Computational Cost: While adapters are used for efficient fine-tuning, the underlying diffusion models can still be computationally intensive.
- Dataset Specificity: The focus on VGGSound-Continuous highlights the model's strength in continuous sound generation. Further evaluation on broader, more diverse datasets might be beneficial.
- Future Work: The authors suggest that their approach can be extended to other controllable audio generation tasks. The framework's ability to integrate various forms of guidance (textual, frame-wise visual) opens avenues for more sophisticated multimodal generation.
Summary updated