A Perspective on Quality Evaluation for AI-Generated Videos

Zhichao Zhang, Wei Sun, Guangtao ZhaiSensors 2025 · DOI

TL;DR

AI-generated videos are rapidly improving, but evaluating their quality is a major challenge. This paper argues that Multimodal Large Language Models (MLLMs) are the future of AI-generated video quality assessment, offering a more holistic and human-aligned evaluation than previous methods.

What Problem Does This Paper Address?

The explosion of AI-generated content (AIGC), particularly text-to-video (T2V) generation, has outpaced our ability to reliably assess the quality of these videos. Traditional video quality assessment (VQA) metrics, designed for compression artifacts, fail to capture the unique failure modes of AIGC, such as inconsistent motion, semantic drift from prompts, and object disocclusion. Existing methods often struggle with generalization and require extensive manual labeling. The core problem is developing robust, scalable, and human-aligned VQA methods for the increasingly sophisticated outputs of AIGC video generation models.

Methods

The paper advocates for a shift towards Multimodal Large Language Models (MLLMs) for AIGC VQA. The core idea is to leverage MLLMs' ability to jointly process and understand information from multiple modalities: vision (frames), language (prompts), and potentially audio and depth.

  • MLLM-based Evaluation: Instead of relying on hand-engineered features or separate models for different quality aspects, MLLMs can process all relevant cues simultaneously. This allows for a holistic assessment of:
  • Spatial fidelity: The visual quality within individual frames.
  • Temporal coherence: The smoothness and consistency of motion across frames.
  • Semantic alignment: How well the video content matches the input text prompt.
  • Technical Approach: The paper describes a "light-tuning" strategy for MLLMs. This involves freezing the vision encoder to preserve low-level perceptual features while fine-tuning the language model to learn a "quality-aware vocabulary." Quality scores are discretized into five categories (Excellent, Good, Fair, Poor, Bad), and the model is trained to predict these tokens. At inference, a weighted sum of probabilities over these tokens provides a final score.
  • Data Efficiency: The authors highlight that MLLMs, combined with techniques like prompt engineering, LORA fine-tuning, and self-distillation, can significantly reduce the need for large, manually annotated datasets, making them more scalable and adaptable.

Datasets

The paper discusses several datasets used for AIGC VQA, categorizing them by their focus and methodology. Key datasets mentioned include:

  • AIGV-Assessor [9]: Focuses on perceptual distortions and uses a large number of videos and expert ratings.
  • T2VBench [58]: Emphasizes temporal dynamics.
  • Human-AGVQA [61]: Specifically targets human artifacts and actions.
  • LGVQ [66] and FETV [64]: Used for empirical evaluation of MLLM-based metrics, focusing on spatial, temporal, and alignment quality.

The paper also reviews existing T2V generation models and their technical specifications, providing context for the evolution of the field.

Key Insights and Ideas

  • MLLMs as the Future: The central argument is that MLLMs are not just an incremental improvement but represent a paradigm shift in AIGC VQA, poised to supersede traditional CNN/ViT architectures.
  • Holistic Evaluation: MLLMs' ability to fuse information from vision, language, and other modalities allows for a more comprehensive assessment that captures the complex interplay of factors contributing to video quality.
  • Bridging the Gap: MLLMs excel at semantic alignment, a critical failure mode for AIGC that traditional metrics struggle with. They can provide natural language rationales for their quality judgments, enhancing interpretability.
  • Data Efficiency and Adaptability: Techniques like prompt engineering and lightweight adapters make MLLM-based VQA systems adaptable to new domains and distortion types with minimal retraining.
  • Beyond Pixel Fidelity: The paper emphasizes that AIGC quality is not just about pixel-level accuracy but also about temporal coherence and semantic faithfulness to the prompt, areas where MLLMs show significant promise.

Key Results

The paper presents empirical results comparing MLLM-based metrics against CNN and VLM baselines on the LGVQ and FETV datasets.

  • Superior Performance: MLLM metrics consistently achieve higher rank correlations (SRCC/KRCC) and linear correlations (PLCC) across spatial, temporal, and alignment quality dimensions compared to the strongest CNN or VLM baselines.
  • Alignment Gains: The advantage of MLLMs is most pronounced for semantic alignment, where they significantly outperform VLMs. For example, on LGVQ, MLLMs closed over one-third of the gap that VLMs had with human judgment.
  • Zero-Shot Capabilities: Alignment quality metrics using MLLMs are shown to be effective even in a zero-shot setting, demonstrating their generalization power.

Limitations and Future Work

  • Hallucinations: MLLMs can still "hallucinate" or provide confident but incorrect responses, even when they don't fully understand the input. While fine-tuning and RLHF can mitigate this, it cannot be entirely eliminated.
  • Interpretability of Scores: While MLLMs can generate continuous quality scores, the interpretability and robustness of this score generation mechanism may still be limited.
  • Edge Cases: Over-reliance on MLLMs might lead to rare but extreme failure cases, though similar issues exist in traditional VQA models.
  • Future Work: The authors envision a roadmap for MLLM-centric VQA pipelines, including:
  • Multimodal fusion at scale: Integrating synchronized visual, textual, and motion tokens for a single-pass evaluation.
  • Data-efficient specialization: Using adapters and prompt engineering for rapid adaptation to new domains.
  • Standardization and deployment: Expecting MLLM cores to become standard in future VQA toolkits.
  • Expanding Modalities: With advancements in sensors, future generative models and evaluation systems will likely support more modalities, leading to more powerful multimodal evaluation.

Summary updated

Abstract

Recent breakthroughs in AI-generated content (AIGC) have transformed video creation, empowering systems to translate text, images, or audio into visually compelling stories. Yet reliable evaluation of these machine-crafted videos remains elusive because quality is governed not only by spatial fidelity within individual frames but also by temporal coherence across frames and precise semantic alignment with the intended message. The foundational role of sensor technologies is critical, as they determine the physical plausibility of AIGC outputs. In this perspective, we argue that multimodal large language models (MLLMs) are poised to become the cornerstone of next-generation video quality assessment (VQA). By jointly encoding cues from multiple modalities such as vision, language, sound, and even depth, the MLLM can leverage its powerful language understanding capabilities to assess the quality of scene composition, motion dynamics, and narrative consistency, overcoming the fragmentation of hand-engineered metrics and the poor generalization ability of CNN-based methods. Furthermore, we provide a comprehensive analysis of current methodologies for assessing AIGC video quality, including the evolution of generation models, dataset design, quality dimensions, and evaluation frameworks. We argue that advances in sensor fusion enable MLLMs to combine low-level physical constraints with high-level semantic interpretations, further enhancing the accuracy of visual quality assessment.
BibTeX
@inproceedings{zhang2025perspective,
  author = {Zhichao Zhang and Wei Sun and Guangtao Zhai},
  title = {{A Perspective on Quality Evaluation for AI-Generated Videos}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25154668},
}
Figure