Abstract
The rapid advancement of AI-generated video (AIGV) models has created a critical need for standardized evaluation frameworks. Current methodologies suffer from fragmented codebases, inconsistent environments, and a lack of alignment with human perception. We present AIGVE-Tool, a unified, modular toolkit designed to streamline the evaluation process through a decoupled architecture. Alongside the toolkit, we introduce AIGVE-Bench, a comprehensive human-annotated dataset comprising 2,430 videos across nine evaluation dimensions. Our analysis reveals a significant "interaction gap" in state-of-the-art models and demonstrates that ensemble-based metric aggregation significantly improves correlation with human judgment.
1. Introduction
The explosion of models such as Sora, CogVideoX, and Hunyuan has outpaced the development of robust measurement tools. Evaluating new models currently involves navigating a "dependency hell" of scattered metrics like FID, FVD, and CLIPScore, which often focus on narrow quality aspects while ignoring physical realism or temporal consistency. Researchers are frequently forced into redundant engineering to re-implement data processing for each new metric. AIGVE-Tool addresses these challenges by providing a standardized interface and a multifaceted benchmark to bridge the gap between automated metrics and human evaluation.
2. The AIGVE-Tool Framework
The core of our contribution is the AIGVE-Tool, built on the modular MMEngine. The framework utilizes a standardized pipeline called AIGVELoop, which decouples data loading from metric logic to ensure flexibility and reproducibility.
The pipeline consists of three primary phases:
- Customizable DataLoaders: Handles video/prompt reading, frame extraction, and resolution normalization.
- Modular Metrics: A registry-based system categorizing metrics into five types: Distribution Comparison (e.g., FVD), Video-Only Neural Networks (e.g., GSTVQA), Vision-Language Similarity (e.g., CLIPSim), Vision-Language Understanding (e.g., VIEScore), and Multi-Faceted frameworks.
- Scores Aggregation: Consolidates results into final evaluation reports.
To improve human alignment, we employ a regression-based ensemble approach. By combining multiple metrics, the Spearman Rank Correlation Coefficient (SRCC) for "Overall" quality improved from 0.24 (best single metric) to 0.35.
3. AIGVE-Bench: A Multifaceted Benchmark
To stress-test state-of-the-art models, we developed AIGVE-Bench. This benchmark includes 500 hand-crafted instructions—split between "global view" landscapes and "close shot" interactions—and 2,430 videos generated by five leading models: Sora, CogVideoX, Hunyuan, Genmo, and Pyramid. We collected 21,870 human labels across nine dimensions, including Technical Quality, Physics, and Interaction Quality.
4. Key Insights and Results
Our evaluation highlights several critical trends in current AIGV performance.
4.1 The Interaction Gap
While models excel at generating aesthetically pleasing natural landscapes (e.g., mountains, oceans), they struggle significantly with Interaction Quality. When prompts require complex object-to-object dynamics—such as a person interacting with tools—physical consistency often breaks down.
4.2 Urban vs. Nature Bias
There is a noticeable performance disparity between natural and urban scenes. Models generally perform better on natural subjects than on complex cityscapes or architecture. This likely stems from training data distributions that are rich in landscapes but lack the high-frequency structural details required for realistic urban environments.
4.3 Model Performance
Hunyuan and Sora emerged as the top performers. Hunyuan achieved the highest mean overall scores across nearly all subjects and evaluation aspects, demonstrating superior consistency and technical quality.
5. Conclusion
AIGVE-Tool provides a necessary foundation for standardizing video generation research. Our findings suggest that the community must move beyond static-quality metrics and focus on the temporal and physical accuracy of complex interactions. Future work will involve scaling the benchmark and exploring non-linear modeling to further refine the alignment between automated metrics and human perception.
Summary updated