Introduction
While AI video generation has seen explosive growth with models like Sora and Gen-2, most existing systems remain "silent," focusing exclusively on visual pixels. The Text to Audible-Video Generation (TAVG) task seeks to move beyond this silent era by generating synchronized audio that matches visual actions—such as the specific screech of tires as a vehicle accelerates.
The TAVG Challenge
The researchers identify two primary bottlenecks in current TAVG research:
- Data Scarcity: A lack of large-scale, high-quality datasets containing synchronized video and audio paired with unified, detailed text descriptions.
- Alignment Issues: Existing "two-stage" methods, which generate video first and then add audio, often fail to achieve semantic harmony between the two modalities.
TAVGBench Dataset
To solve the data problem, the authors introduced TAVGBench, the first large-scale benchmark for this task. It contains 1.7 million clips (approximately 11,800 hours) sourced from YouTube.
To annotate this massive dataset, they developed an automatic "coarse-to-fine" pipeline. This system uses BLIP-2 for video captions and WavCaps for audio captions, then employs ChatGPT to merge these into a single, coherent, human-like description that captures both the sights and sounds of a scene.
TAVDiffusion Architecture
The core of the technical contribution is TAVDiffusion, a two-stream latent diffusion model designed for simultaneous generation.
- Video Stream: Based on Stable Diffusion with added temporal layers to handle motion.
- Audio Stream: Generates mel spectrograms which are then converted to waveforms.
- Multimodal Interaction: The model uses cross-attention to share information between audio and video features.
- Explicit Audio-Visual Alignment (EAS): This strategy uses contrastive learning to ensure that features from the same clip are mathematically closer in the latent space, fostering better synchronization.
Evaluation and Results
The paper introduces AVHScore, a metric that uses ImageBind to project audio and video into a shared space, allowing for a quantitative measure of "harmony."
TAVDiffusion significantly outperformed existing two-stage baselines:
- Visual Quality: On TAVGBench, it achieved an FVD (Frechet Video Distance) of 516.56, compared to 777.54 for the best baseline.
- Alignment: The model reached an AVHScore of 23.35, nearly doubling the performance of the best baseline (12.28).
- Zero-Shot Performance: The model maintained superior performance even on unseen data (FAVDBench), indicating strong generalization capabilities.
Conclusion
TAVGBench and TAVDiffusion demonstrate that generating audio and video simultaneously leads to superior alignment and quality compared to fragmented pipelines. By utilizing LLMs as a linguistic bridge for data annotation and implementing explicit alignment strategies, this work provides a robust foundation for future research in audible video generation. Future iterations may move toward Multimodal Diffusion Transformers (DiT) to further improve scalability.
Summary updated