Abstract
The rapid advancement of generative video models, such as Sora and Movie Gen, has outpaced the development of robust evaluation frameworks. This survey explores the emerging field of AI-Generated Video Evaluation (AIGVE), categorizing current methodologies into perceptual quality and instructional alignment. We discuss the transition from traditional metrics to AI-based judges and highlight the significant challenges remaining in modeling physical commonsense and ensuring safety.
Introduction
The field of video generation is currently experiencing a "Cambrian explosion," with new models emerging almost monthly. Unlike static images, video evaluation is exponentially more complex due to temporal dynamics, physical consistency, and narrative flow. Traditional Video Quality Assessment (VQA) metrics, designed for compression artifacts or transmission noise, fail to capture AI-specific "hallucinations"—such as a person growing a third arm or objects defying gravity.
The Dual-Pillar Framework
AIGVE is built on two primary pillars:
- Perceptual Quality: Assessing how "good" the video looks visually, focusing on texture, resolution, and temporal stability.
- Instructional Alignment: Measuring how well the generated content follows the user's prompt. A video can be visually stunning but fail if it ignores specific requests, such as a "red hat" or a specific action.
Methodological Approaches
Researchers generally follow two paths for evaluation:
- Metric Collection: An ensemble approach using specialized tools like DINO for subject consistency, CLIP for text-video alignment, and RAFT for motion intensity.
- Modeling Evaluation: The "AI judging AI" approach. Models like VideoScore or VLM-based judges (e.g., GPT-4V) are trained to provide human-like scores or qualitative feedback, moving beyond simple mathematical formulas like PSNR.
Benchmarking and Datasets
New benchmarks have emerged to address the unique failures of AI:
- VBench & EvalCrafter: Provide comprehensive hierarchical dimensions for general quality.
- VIDEOPHY: Specifically tests "physical commonsense"—whether a video obeys laws like gravity and object permanence.
- T2VSafetyBench: Evaluates ethical alignment, checking for violence, discrimination, or inappropriate content.
The Physics Gap and Causal Reality
One of the most significant hurdles in AIGVE is the "Physics Gap." Current models are proficient at rendering textures but often struggle with "causal physics." The VIDEOPHY benchmark reveals that even advanced models achieve accurate physical commonsense in only about 19.7% of cases. These failures manifest as objects shattering prematurely or liquids behaving unnaturally.
Key Results and Future Frontiers
Modeling-based evaluators have reached a Spearman correlation of 77.1 with human feedback, indicating a shift toward automated, sophisticated "judges." However, compositional changes—where objects change state or relationship over time—remain a major weakness, with success rates often below 20%.
Future research must focus on:
- Interpretability: Moving away from "black box" scores toward "Chain-of-Thought" feedback.
- Efficiency: Developing lightweight evaluators that do not require massive compute.
- World Models: Integrating a deeper understanding of physical reality into the evaluation process.
- Ethical Embedding: Ensuring safety is a core component of the deployment pipeline.
Summary updated