Abstract
Current Video-to-Audio (V2A) models often fail when the sound-making object is partially off-screen—a common occurrence in professional filmmaking known as "cinematic language." This paper introduces a self-distillation framework that trains a model to infer full audio-visual context from partial visual cues. By aligning features from simulated cinematic shots with those from full-view videos, the proposed method significantly improves the quality and synchronization of generated sound in complex cinematic scenarios.
Introduction
In the world of film production, "Foley" is the art of creating sound effects—such as footsteps, rustling clothes, or sword clashes—to match the visuals. While AI-driven V2A models have made significant strides in automating this process, they possess a notable "blind spot": cinematic language.
Professional directors frequently use techniques like close-ups (where only a portion of an instrument might be visible) or dynamic camera movements (where an object enters or leaves the frame). Standard V2A models are typically trained on "clean" datasets where the object making the noise is front and center. When faced with a close-up of a pianist's hands where the piano itself is barely visible, these models often become "puzzled," resulting in audio that is either out of sync or semantically irrelevant. This research addresses this gap by enabling models to "fill in the blanks" of incomplete visual information.
Methodology
The authors propose a Teacher-Student self-distillation framework designed to bridge the gap between standard training data and cinematic reality without the need for expensive, specialized datasets.
1. Simulating Cinematic Language
Instead of collecting new footage, the framework applies specific transformations ($f$) to existing "full-view" videos to simulate cinematic shots:
- Close-ups ($f_{cu}$): Random spatial cropping mimics tight framing where only a portion of the subject is visible.
- Camera Movements ($f_{cm}$): Temporal and spatial shifts simulate panning or tracking shots where the subject moves relative to the frame.
2. The Teacher-Student Setup
The framework utilizes a dual-encoder architecture:
- The Teacher: A frozen, pre-trained video encoder that sees the original, full-view video, providing the "ground truth" context.
- The Student: A trainable encoder that sees the "cinematic" (partial) version of the same video.
3. Feature Alignment and Generation
The core objective is to force the Student's output features to match the Teacher's features. By minimizing the distance (using MSE and Cosine Similarity) between these two feature sets, the Student learns to map incomplete visual information to the same semantic space as the full scene. These refined features are then fed into a Latent Diffusion Model (LDM)—specifically building on the Diff-Foley architecture—to generate the final audio.
Experimental Setup
The model was evaluated using a variety of datasets to test its robustness:
- VGGSound: The primary large-scale dataset (309 sound categories) used for training.
- VGG-CU (Close-Up) & VGG-CM (Camera Movement): New test sets created by the authors to evaluate performance under specific partial visibility conditions.
- Real-world Cinematic Set: A curated collection of 31 YouTube clips featuring professional cinematic techniques (zooms, pans, transitions) used for human evaluation.
Key Insights and Results
Learning from Incompleteness
The most powerful insight of this work is that a model does not need to see an entire object to identify its sound; it simply needs to be trained to recognize the relevant clues. By aligning partial features with full features, the model learns the "logic" of cinematic framing.
Quantitative Performance
The proposed method outperformed the state-of-the-art (SOTA) baseline, Diff-Foley, across all primary metrics:
- Alignment Accuracy: On the VGG-CU set, accuracy increased from 74.8% to 85.0%. On the VGG-CM set, it rose from 72.5% to 81.8%.
- Audio Quality: The Frechet Audio Distance (FAD)—a measure of audio realism—improved significantly, dropping from 8.926 to 7.825 in close-up scenarios.
Generalization and Human Preference
Interestingly, training primarily on simulated close-ups also improved performance on camera movements, suggesting the model developed a generalized robustness to partial visibility. Furthermore, in a study with 20 participants watching real-world cinematic clips, 74.5% of users preferred the audio generated by the authors' method over the baseline.
Conclusion
This paper demonstrates that self-distillation is an effective strategy for teaching V2A models to interpret the nuances of cinematic language. By simulating partial visibility during training, the model learns to generate high-quality, synchronized audio even when the visual source is obscured. Future work may expand this approach to other cinematic elements, such as extreme lighting conditions or rapid montage editing, further closing the gap between AI generation and professional film production.
Summary updated