Artifact-Aware Evaluation for High-Quality Video Generation

Chen Zhu, Jiashu Zhu, Yanxun Li, Meiqi Wu, Bingze Song, Chubin Chen, Jiahong Wu, Xiangxiang Chu, Yangang WangarXiv 2026 · arXiv · OpenReview

TL;DR

This paper tackles the challenge of evaluating the quality of AI-generated videos, which often suffer from subtle but noticeable artifacts. They introduce a new, human-centric evaluation framework and a large dataset (GenVID) to train a model called DVAR, which can precisely detect and categorize these artifacts, enabling better filtering of low-quality content.

What Problem Does This Paper Address?

The rapid progress in AI video generation has led to an explosion of synthetic content. While these models are impressive, they frequently introduce visual glitches or "artifacts" – things like flickering textures, unnatural movements, or shaky camera work. Current methods for evaluating video quality often provide a single, coarse score, which doesn't tell us what is wrong or where. This makes it hard for platforms to automatically filter out bad videos or for researchers to understand and fix specific generation issues. The paper aims to bridge this gap by providing a more detailed, human-aligned way to assess video quality, focusing on the specific artifacts that viewers find objectionable.

Methods

The core of the paper's contribution lies in its novel evaluation framework and the DVAR model.

  • Artifact-Centric Evaluation Protocol: Instead of a single score, they propose evaluating videos along three perceptual axes: Appearance, Motion, and Camera. These axes are further broken down into a taxonomy of 10 specific artifact categories (e.g., "Blurred Visuals," "Physically Implausible Movements," "Shaky Camera Motion"). This structured approach mimics how humans naturally perceive and judge video quality.
  • GenVID Dataset: To train models for this detailed evaluation, they created GenVID, a massive dataset of 80,000 generated videos from various state-of-the-art models. Crucially, each video is meticulously annotated with labels for the 10 artifact categories.
  • Flow-Magnitude-Guided Dynamic Frame Sampling (FMG-DFS): Processing every single frame of a video is computationally expensive. FMG-DFS is a smart sampling technique that identifies frames with high motion intensity (using optical flow) because these are more likely to contain artifacts. This makes the evaluation process more efficient without sacrificing important details.
  • DVAR (Dense Video Artifact Recognition): This is the proposed framework that combines the FMG-DFS sampling with a fine-grained artifact detection model. By training a multimodal large language model (MLLM) on the GenVID dataset, DVAR learns to identify and classify the 10 artifact categories with high precision.

Datasets

  • GenVID: A newly introduced, large-scale dataset comprising 80,000 generated videos. It's annotated with 10 distinct artifact categories across Appearance, Motion, and Camera dimensions, serving as a benchmark for artifact detection models.

Key Insights and Ideas

  • Human Perception is Key: The paper strongly advocates for moving beyond generic quality scores to a more granular, human-aligned evaluation that focuses on specific, perceivable artifacts.
  • Motion is a Hotspot for Artifacts: The insight that segments with high motion dynamics are more prone to artifacts is crucial. This led to the development of the efficient FMG-DFS sampling strategy.
  • Large-Scale, Annotated Data is Essential: The creation of GenVID highlights the need for substantial, well-annotated datasets to train robust models capable of understanding nuanced visual defects in AI-generated content.
  • MLLMs Can Be Fine-Tuned for Specific Tasks: By fine-tuning existing MLLMs on GenVID, the authors demonstrate their potential for specialized tasks like detailed artifact recognition, significantly outperforming general-purpose models.

Key Results

  • DVAR significantly outperforms existing SOTA models in video artifact recognition, achieving an accuracy of 80.0% for the "All" categories combined.
  • In the "Appearance" dimension, DVAR shows a substantial improvement in accuracy compared to the best-performing baseline.
  • The FMG-DFS sampling strategy demonstrates superior efficiency in capturing artifact frames compared to uniform sampling, leading to improved performance.
  • Increasing model parameter size yielded only minor improvements, suggesting that the gains are largely due to the quality of the GenVID dataset and the fine-tuning process.
  • When used as a filtering module, DVAR effectively suppresses low-quality video outputs while preserving high-quality generations.

Limitations and Future Work

  • Computational Cost: While FMG-DFS improves efficiency, processing and evaluating large numbers of videos can still be computationally intensive.
  • Generalization to Real-World UGC: The paper focuses on generated videos. While the insights are valuable, further work might be needed to see how well DVAR generalizes to the diverse and often more complex artifacts found in user-generated content.
  • Future Directions: The authors suggest that their framework can be extended to other types of generative models and applied to real-world moderation pipelines. Further research could explore more sophisticated artifact detection techniques and expand the artifact taxonomy.

Summary updated

Abstract

With the rapid advancement of video generation techniques, evaluating and auditing generated videos has become increasingly crucial. Existing approaches typically offer coarse video quality scores, lacking detailed localization and categorization of specific artifacts. In this work, we introduce a comprehensive evaluation protocol focusing on three key aspects affecting human perception: Appearance, Motion, and Camera. We define these axes through a taxonomy of 10 prevalent artifact categories reflecting common generative failures observed in video generation. To enable robust artifact detection and categorization, we introduce GenVID, a large-scale dataset of 80k videos generated by various state-of-the-art video generation models, each carefully annotated for the defined artifact categories. Leveraging GenVID, we develop DVAR, a Dense Video Artifact Recognition framework for fine-grained identification and classification of generative artifacts. Extensive experiments show that our approach significantly improves artifact detection accuracy and enables effective filtering of low-quality content.
BibTeX
@misc{zhu2026artifact,
  author = {Chen Zhu and Jiashu Zhu and Yanxun Li and Meiqi Wu and Bingze Song and Chubin Chen and Jiahong Wu and Xiangxiang Chu and Yangang Wang},
  title = {{Artifact-Aware Evaluation for High-Quality Video Generation}},
  year = {2026},
  eprint = {2601.20297},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.20297},
}
Figure