TL;DR
The Audio-Video Vector Alignment (AVVA) framework demonstrates that when training multimodal foundation models, data quality beats data quantity. By using a pipeline of Large Language Models (LLMs) to curate and filter out poorly aligned audio-visual data, AVVA achieves state-of-the-art cross-modal retrieval performance using just 192 hours of training data—a 30x reduction compared to previous uncurated approaches.
What Problem Does This Paper Address?
When humans watch a video, we instinctively connect visual cues with their corresponding sounds without needing a textual description to bridge the two. However, most current multimodal AI systems (like CLIP or CLAP) rely heavily on text captions to learn these relationships.
While recent "text-free" audio-visual models attempt to map audio directly to video, they typically require massive datasets (often upwards of 5,000 hours) to learn effectively. This is because raw internet video data is incredibly noisy; a video of a kitchen might have unrelated background music playing, or a camera might show a speaker long after they've finished talking. The authors of this paper ask a fundamental question: Can we build a highly efficient audio-video foundation model by strictly filtering for high-quality, deeply aligned data?
Methods
The AVVA framework splits the problem into two distinct phases: LLM-based data curation and language-free model training.
1. The Multimodal Reasoning Engine (MRE)
Instead of relying on simple temporal synchronization, the authors designed the MRE to evaluate the semantic and contextual coherence of audio-video pairs.
- Caption Generation: For every 3-second video clip,
LLaVA-Next-Videogenerates a visual description, whileLTU-ASgenerates an audio description. - LLM Scoring: These two text descriptions are fed into
Mistral-7B Instruct v0.3, which rates the alignment from 0 to 10 across five distinct metrics: Temporal Alignment, Spatial Coherence, Contextual Relevance, Physical Causality, and Sound Source Visibility. - Filtering: Clips falling below a specific average threshold are discarded, leaving only highly synchronized training data.
2. Model Architecture
Once the data is curated, the actual AVVA model is trained without any text mediation:
- Backbones: It pairs a frozen
Whispermodel (for audio) with a frozenDINOv2model (for video). - Fusion: Features are passed through learnable aligner layers (MLPs) and a Bidirectional Cross-Modal Attention mechanism to force the two streams to interact dynamically.
- Objective: The model is optimized using an InfoNCE contrastive loss function to pull matching audio-video vectors closer together in the embedding space.
Datasets
The authors aggregated and curated data from a massive variety of audiovisual sources, including:
- Egocentric/Action: Epic-Kitchens, HowTo100M
- General/Ambient: AudioSet, AudioCaps, VGGSound, AVE, HD-VILA-100M
- Music: Music-MIT
- Instructional/Task-oriented: VALOR
Evaluations for cross-modal retrieval were performed on the standard benchmarks: AudioCaps, VALOR, and VGGSound.
Key Insights and Ideas
- The "Quality over Quantity" Trade-off: More data isn't always better. The ablation studies show a clear trend: as the curation threshold increases (meaning more data is thrown away, leaving a smaller but cleaner dataset), downstream retrieval performance actually improves.
- LLMs as Advanced Filters: Traditional data filtering relies on simple heuristics (like text keyword matching). AVVA proves that LLMs can reason about abstract concepts like physical causality (e.g., matching the sound of shattering glass to the visual of a breaking window) to perform highly sophisticated data cleaning.
- Speech Backbones for General Audio: Interestingly, using Whisper—a model trained primarily on speech—as a frozen audio backbone yields excellent generalization for general environmental and ambient sounds when properly aligned with a strong visual model like DINOv2.
Key Results
- 30x Data Efficiency: AVVA achieved audio-to-video retrieval performance comparable to DenseAV, and significantly outperformed it in video-to-audio retrieval, while using only 192 hours of curated data compared to DenseAV’s 5,800+ hours.
- Curation Gains: Training on the curated subset yielded massive performance jumps over training on the full, uncurated datasets. For instance, on the VALOR dataset, audio-to-video retrieval saw a 22.67% increase in Top-1 accuracy purely due to the curation process.
- Temporal Precision: To verify that the model truly learned temporal alignment, the authors ran simulations shifting the audio track out of sync. The model's cosine similarity sharply peaked at exactly 0 seconds of shift, proving it is highly sensitive to real-time synchronization.
Limitations and Future Work
The primary bottleneck of the AVVA framework is the computational cost of curation. Running three separate LLMs (LLaVA, LTU-AS, and Mistral) to score every single 3-second clip adds roughly 6 seconds of preprocessing time per GPU per segment.
The authors note that future work will focus on making this curation pipeline more efficient, perhaps by distilling the reasoning capabilities into smaller, faster edge models, or exploring more lightweight joint-reasoning architectures.
Summary updated