AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment

Yuqin Cao, Xiongkuo Min, Yixuan Gao, Wei Sun, Guangtao Zhai

Introduction

The rapid advancement of generative AI has enabled sophisticated Video-to-Audio (VTA) tools capable of synthesizing soundscapes for silent, AI-generated videos. However, these tools often produce outputs that feel "uncanny"—audio that may be technically clear but is semantically mismatched or poorly synchronized with the visual action. Traditional Audio-Visual Quality Assessment (AVQA) metrics, primarily designed to detect technical artifacts like compression noise or transmission lag, fail to capture the nuances of semantic consistency and perceptual realism required for AI-generated audio-visual (AGAV) content.

To address this gap, we introduce AGAV-Rater, a novel Large Multimodal Model (LMM) framework specifically optimized for evaluating the quality of AGAV content. By leveraging a specialized two-stage training pipeline and a new large-scale dataset, AGAV-Rater achieves state-of-the-art performance in both quality scoring and candidate selection.

The AGAVQA-3k Dataset

A significant hurdle in AGAV research has been the lack of comprehensive benchmarks. This work introduces AGAVQA-3k, the first large-scale dataset dedicated to this domain. The dataset comprises 3,382 AGAV samples generated by 16 distinct VTA methods, categorized into two subsets:

  • AGAVQA-MOS: Contains 3,088 samples with over 9,000 human-assigned Mean Opinion Scores (MOS) across three dimensions: audio quality, audio-visual consistency, and overall quality.
  • AGAVQA-Pair: Consists of 294 samples organized into 75 groups, designed to evaluate a model's ability to select the superior audio track from multiple candidates.

Methodology: AGAV-Rater

AGAV-Rater is built upon the VideoLLaMA2 backbone. Its training methodology addresses the scarcity of human-labeled data through a two-stage approach:

1. Pre-training via Text-Defined Levels

To instill a foundational understanding of audio quality, the authors generated approximately 51,000 instruction-response pairs using real-world datasets. They employed a "reversed audio" strategy, where high-quality audio tracks were reversed or swapped with mismatched captions to simulate "bad" quality. This allowed the model to learn the difference between "excellent" and "poor" content at scale without requiring manual annotation.

2. Fine-tuning via Numerical Regression

Unlike standard LMMs that output categorical text tokens (e.g., "Good"), AGAV-Rater incorporates a regression head attached to the LLM's final hidden states. This allows the model to predict precise numerical scores. The training objective utilizes Pearson Linear Correlation Coefficient (PLCC) loss to align the model’s predictions with human perception nuances.

Furthermore, the model employs a multi-dimensional instruction strategy. By prompting the model to evaluate specific attributes—such as audio-visual synchronization and semantic matching—before providing an overall score, the framework mimics human cognitive processes, leading to more robust holistic judgments.

Key Results and Performance

AGAV-Rater demonstrates significant improvements over existing benchmarks and proprietary models:

  • State-of-the-Art Scoring: On the AGAVQA-MOS benchmark, the model achieved a Spearman Rank Correlation Coefficient (SRCC) of 0.7909 for audio quality and 0.7553 for A/V consistency.
  • Superior Selection Accuracy: In the AGAVQA-Pair selection task, AGAV-Rater achieved an accuracy of 0.78, vastly outperforming proprietary models like GPT-4o (approx. 0.30) and Gemini 1.5 Pro (0.23).
  • Practical Utility: In subjective user preference tests, 80% of participants preferred the VTA outputs selected by AGAV-Rater over default generative outputs.
  • Efficiency: The model is optimized for real-world deployment, processing a 3-second video in approximately 157ms on a single NVIDIA RTX 4090 GPU.

Conclusion

AGAV-Rater represents a significant step forward in the automated evaluation of generative media. By combining synthetic pre-training with a regression-based fine-tuning approach, the model captures the complex relationship between audio and video more effectively than general-purpose LMMs. Future work will explore using AGAV-Rater as a supervisory signal to provide direct feedback loops for training the next generation of Video-to-Audio models, potentially moving beyond realism checks to more subjective assessments like musical style and emotional resonance.

Summary updated

BibTeX
@inproceedings{cao2025agav,
  author = {Yuqin Cao and Xiongkuo Min and Yixuan Gao and Wei Sun and Guangtao Zhai},
  title = {{AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=v0Ui6SJ1d4},
}
Figure