VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation

Saksham Singh Kushwaha, Yapeng Tian

Introduction

VinTAGe is a novel flow-based transformer model designed to generate "holistic" audio by simultaneously conditioning on both video and text prompts. While previous models often struggle to balance visual synchronization with semantic context, VinTAGe employs a teacher-student framework to ensure generated sounds are perfectly aligned with onscreen actions while incorporating offscreen elements described in text.

Problem Statement: The Blind Spots of Uni-modal Generation

Automated sound design, or Foley, typically addresses two distinct tasks: Text-to-Audio (T2A) and Video-to-Audio (V2A). However, both modalities have inherent limitations. T2A models can generate complex soundscapes from descriptive prompts (e.g., "a dog barks while a vacuum hums") but lack the temporal information to synchronize sounds with specific visual events. Conversely, V2A models excel at timing but are restricted to generating sounds for objects visible within the frame. If a sound source is off-camera, a V2A model will likely omit it.

The challenge lies in holistic audio generation: creating a single audio track that is both visually synchronized (onscreen) and semantically complete (offscreen). Previous joint conditioning approaches often suffered from modality bias, where the model would prioritize one input over the other, leading to poor alignment or incomplete soundscapes.

Figure 1: Comparison of V2A, T2A, and the proposed holistic approach.

Proposed Method: The VinTAGe Framework

The VinTAGe framework is built upon three primary architectural pillars:

  1. Visual-Text (VT) Encoder: Rather than simple feature concatenation, this module utilizes a gated cross-attention mechanism to facilitate interaction between text and video features. To capture temporal dynamics, the encoder incorporates mean optical flow and frame indices, providing a "temporal energy map" that guides the model on when motion-related sounds should occur.
  2. Joint VT-SiT Model: The core generative engine is based on Scalable Interpolant Transformers (SiT), a class of flow-matching models. Flow-matching offers a stable alternative to traditional diffusion by "straightening" the probability path from noise to data, often resulting in more efficient training.
  3. Teacher-Guided Alignment: To mitigate modality bias, VinTAGe employs a teacher-student framework. Pretrained, high-performing uni-modal models (one for T2A and one for V2A) serve as "teachers." During training, the joint VinTAGe model is regularized to ensure its predictions do not deviate significantly from the specialized teachers' outputs when provided with single-modality inputs. This forces the model to leverage both text and video information effectively.
Figure 2: Detailed architecture of VinTAGe, including the VT-encoder (A), the VT-SiT block (B), and the teacher-guided training framework (C).

Experimental Setup and Datasets

The model was rigorously evaluated using:

  • VGGSound: A standard dataset comprising approximately 200,000 video-audio pairs.
  • VinTAGe-Bench: A new benchmark specifically designed to test for offscreen sounds. It includes 636 pairs across 14 classes, where videos are paired with prompts containing both onscreen and additional offscreen sound descriptions (e.g., a video of a bird chirping with a text prompt mentioning a toilet flushing).

Key Insights

  • Optical Flow as a Temporal Guide: Calculating the average magnitude of optical flow provides an effective, low-compute shortcut for identifying sound onsets, significantly aiding temporal alignment.
  • Addressing Modality Bias: The use of uni-modal teachers as regularizers is critical. It prevents the model from collapsing into a "video-only" or "text-only" generator, ensuring a balanced integration of both inputs.
  • Information Hierarchy: Holistic generation requires the model to learn a hierarchy: video primarily dictates timing, while text dictates semantic content.

Results and Discussion

VinTAGe achieved state-of-the-art (SOTA) performance on both standard and holistic benchmarks:

  • VGGSound: Achieved a Frechet Audio Distance (FAD) of 6.65, outperforming models like Diff-Foley (9.87) and FoleyCrafter (9.17).
  • VinTAGe-Bench: Demonstrated superior performance with an FAD of 3.05 (compared to 4.05 for the next best model) and higher "Concept Accuracy" in including offscreen sounds.
  • Subjective Evaluation: Human listening tests (MOS) consistently ranked VinTAGe higher for both overall audio quality and temporal synchronization.

Limitations and Future Work

While VinTAGe represents a significant advancement, several challenges remain. The model currently performs best on relatively simple scenes; generating audio for complex, multi-layered cinematic environments is a target for future work. Additionally, the model's performance is tied to the quality of its pretrained encoders (CLIP and FLAN-T5). Finally, extending the generation capabilities from 10-second clips to longer-form video remains an open research area.

Reference: Kushwaha, S. S., & Tian, Y. (2024). VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Summary updated

BibTeX
@inproceedings{kushwaha2025vintage,
  author = {Saksham Singh Kushwaha and Yapeng Tian},
  title = {{VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01263},
}
Figure