Quality Over Quantity? LLM-Based Curation for a Data-Efficient Audio-Video Foundation Model

Ali Vosoughi, Dimitra Emmanouilidou, Hannes GamperarXiv 2025 · arXiv

TL;DR

The Audio-Video Vector Alignment (AVVA) framework demonstrates that when training multimodal foundation models, data quality beats data quantity. By using a pipeline of Large Language Models (LLMs) to curate and filter out poorly aligned audio-visual data, AVVA achieves state-of-the-art cross-modal retrieval performance using just 192 hours of training data—a 30x reduction compared to previous uncurated approaches.

What Problem Does This Paper Address?

When humans watch a video, we instinctively connect visual cues with their corresponding sounds without needing a textual description to bridge the two. However, most current multimodal AI systems (like CLIP or CLAP) rely heavily on text captions to learn these relationships.

While recent "text-free" audio-visual models attempt to map audio directly to video, they typically require massive datasets (often upwards of 5,000 hours) to learn effectively. This is because raw internet video data is incredibly noisy; a video of a kitchen might have unrelated background music playing, or a camera might show a speaker long after they've finished talking. The authors of this paper ask a fundamental question: Can we build a highly efficient audio-video foundation model by strictly filtering for high-quality, deeply aligned data?

Methods

The AVVA framework splits the problem into two distinct phases: LLM-based data curation and language-free model training.

Fig. 1: Overview of the proposed audiovisual alignment approach. (a) Our method’s data curation stage uses multimodal reasoning to retain only highly aligned data. It uses LLaVA- Next-Video [10], [11] for video reasoning, LTU-AS [12] for audio processing, and Mistral [13] for alignment scoring. (b) AVVA employs Whisper (audio), DINOv2 (video backbone), without the need for textual mediation during training.

1. The Multimodal Reasoning Engine (MRE)

Instead of relying on simple temporal synchronization, the authors designed the MRE to evaluate the semantic and contextual coherence of audio-video pairs.

  • Caption Generation: For every 3-second video clip, LLaVA-Next-Video generates a visual description, while LTU-AS generates an audio description.
  • LLM Scoring: These two text descriptions are fed into Mistral-7B Instruct v0.3, which rates the alignment from 0 to 10 across five distinct metrics: Temporal Alignment, Spatial Coherence, Contextual Relevance, Physical Causality, and Sound Source Visibility.
  • Filtering: Clips falling below a specific average threshold are discarded, leaving only highly synchronized training data.

2. Model Architecture

Once the data is curated, the actual AVVA model is trained without any text mediation:

  • Backbones: It pairs a frozen Whisper model (for audio) with a frozen DINOv2 model (for video).
  • Fusion: Features are passed through learnable aligner layers (MLPs) and a Bidirectional Cross-Modal Attention mechanism to force the two streams to interact dynamically.
  • Objective: The model is optimized using an InfoNCE contrastive loss function to pull matching audio-video vectors closer together in the embedding space.

Datasets

The authors aggregated and curated data from a massive variety of audiovisual sources, including:

  • Egocentric/Action: Epic-Kitchens, HowTo100M
  • General/Ambient: AudioSet, AudioCaps, VGGSound, AVE, HD-VILA-100M
  • Music: Music-MIT
  • Instructional/Task-oriented: VALOR

Evaluations for cross-modal retrieval were performed on the standard benchmarks: AudioCaps, VALOR, and VGGSound.

Key Insights and Ideas

  • The "Quality over Quantity" Trade-off: More data isn't always better. The ablation studies show a clear trend: as the curation threshold increases (meaning more data is thrown away, leaving a smaller but cleaner dataset), downstream retrieval performance actually improves.
  • LLMs as Advanced Filters: Traditional data filtering relies on simple heuristics (like text keyword matching). AVVA proves that LLMs can reason about abstract concepts like physical causality (e.g., matching the sound of shattering glass to the visual of a breaking window) to perform highly sophisticated data cleaning.
  • Speech Backbones for General Audio: Interestingly, using Whisper—a model trained primarily on speech—as a frozen audio backbone yields excellent generalization for general environmental and ambient sounds when properly aligned with a strong visual model like DINOv2.

Key Results

  • 30x Data Efficiency: AVVA achieved audio-to-video retrieval performance comparable to DenseAV, and significantly outperformed it in video-to-audio retrieval, while using only 192 hours of curated data compared to DenseAV’s 5,800+ hours.
  • Curation Gains: Training on the curated subset yielded massive performance jumps over training on the full, uncurated datasets. For instance, on the VALOR dataset, audio-to-video retrieval saw a 22.67% increase in Top-1 accuracy purely due to the curation process.
Fig. 4: Audio-to-video model performance over hours of training data, as determined by varying the selection of the MRE score threshold, shown for Top-k = {1, 3, 10} accuracies.
  • Temporal Precision: To verify that the model truly learned temporal alignment, the authors ran simulations shifting the audio track out of sync. The model's cosine similarity sharply peaked at exactly 0 seconds of shift, proving it is highly sensitive to real-time synchronization.

Limitations and Future Work

The primary bottleneck of the AVVA framework is the computational cost of curation. Running three separate LLMs (LLaVA, LTU-AS, and Mistral) to score every single 3-second clip adds roughly 6 seconds of preprocessing time per GPU per segment.

The authors note that future work will focus on making this curation pipeline more efficient, perhaps by distilling the reasoning capabilities into smaller, faster edge models, or exploring more lightweight joint-reasoning architectures.

Summary updated

Abstract

Integrating audio and visual data for training multimodal foundational models remains a challenge. The Audio-Video Vector Alignment (AVVA) framework addresses this by considering AV scene alignment beyond mere temporal synchronization, and leveraging Large Language Models (LLMs) for data curation. AVVA implements a scoring mechanism for selecting aligned training data segments. It integrates Whisper, a speech-based foundation model, for audio and DINOv2 for video analysis in a dual-encoder structure with contrastive learning on AV pairs. Evaluations on AudioCaps, VALOR, and VGGSound demonstrate the effectiveness of the proposed model architecture and data curation approach. AVVA achieves a significant improvement in top-k accuracies for video-to-audio retrieval on all datasets compared to DenseAV, while using only 192 hrs of curated training data. Furthermore, an ablation study indicates that the data curation process effectively trades data quality for data quantity, yielding increases in top-k retrieval accuracies on AudioCaps, VALOR, and VGGSound, compared to training on the full spectrum of uncurated data.
BibTeX
@misc{vosoughi2025quality,
  author = {Ali Vosoughi and Dimitra Emmanouilidou and Hannes Gamper},
  title = {{Quality Over Quantity? LLM-Based Curation for a Data-Efficient Audio-Video Foundation Model}},
  year = {2025},
  eprint = {2503.09205},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.09205},
}
Figure