Abstract
Evaluating the quality of AI-generated videos remains a significant challenge due to the subtle, localized nature of generative artifacts. Traditional metrics often fail to capture fine-grained defects such as anatomical distortions or physics violations. This paper introduces FingER (Fine-grained Entity-level Reasoning), a novel evaluation framework that decomposes video quality assessment into entity-specific questions. By leveraging a reasoning-capable Multi-modal Large Language Model (MLLM) trained via Group Relative Policy Optimization (GRPO), FingER achieves state-of-the-art performance on major benchmarks while demonstrating remarkable data efficiency.
1. Introduction
As video generation models like Sora, Kling, and Luma advance, the gap between visual fidelity and structural consistency has become a critical bottleneck. While a generated video may appear high-quality at a global level, closer inspection often reveals "hallucinations"—such as a person with six fingers, inconsistent reflections, or objects teleporting across frames.
Current Video Quality Assessment (VQA) methods generally fall into two categories: feature-based metrics (e.g., FVD) that lack semantic depth, and MLLM-based scorers (e.g., VideoScore) that provide a single scalar value. Both approaches struggle with fine-grained reasoning. Even advanced models like GPT-4o frequently overlook localized artifacts if the overall scene is realistic. Furthermore, most MLLMs suffer from a domain gap, as they are primarily trained on natural videos rather than the specific types of artifacts unique to AI-generated content.
2. The FingER Framework
The FingER framework addresses these limitations by shifting the focus from global scoring to entity-level reasoning. The process operates in three distinct stages:
2.1 Entity-level Question Generation
Rather than asking for a general quality score, the system utilizes an LLM (GPT-4o) to analyze the input prompt and extract specific entities. It then generates targeted questions across five critical dimensions: Visual Quality, Temporal Consistency, Factual Consistency, Dynamic Degree, and Text-to-Video (T2V) Alignment. For example, instead of "Is the video good?", the system asks, "Does the girl's hand deform as she walks?"
2.2 Reasoning Model Training
The core of FingER is a reasoning model based on Qwen2.5-VL. To enable the model to "think" before scoring, the researchers employed Group Relative Policy Optimization (GRPO), a reinforcement learning technique. The training follows a "cold-start" strategy: the model is first fine-tuned on high-quality reasoning chains and then optimized via RL to improve accuracy and formatting. This forces the model to generate a Chain-of-Thought (CoT) before arriving at a final assessment.
2.3 Hierarchical Scoring
The model produces a binary "Yes/No" answer accompanied by a detailed reasoning block. These responses are converted into probabilities and weighted to generate a final, interpretable score that reflects both the presence of artifacts and the degree of alignment with the original prompt.
3. Data and Benchmarking
To support this framework, the authors curated FingER-Instruct-60k, a high-quality dataset featuring 3,300 AI-generated videos from top-tier models. This dataset includes 60,000 fine-grained QA annotations, each paired with a detailed reasoning process explaining the logic behind the evaluation.
FingER was evaluated against GenAI-Bench and MonetBench, two of the most rigorous public benchmarks for AI video evaluation.
4. Key Insights and Results
- State-of-the-Art Performance: FingER outperformed existing methods by a relative margin of 11.8% on GenAI-Bench and 5.5% on MonetBench.
- Data Efficiency: Remarkably, FingER achieved these results using only 3,300 videos, whereas previous models often required over 600,000 samples. This suggests that the quality and granularity of reasoning are more critical than raw dataset volume.
- Superior Defect Detection: Qualitative analysis demonstrates that FingER identifies subtle defects—such as missing reflections in water or minor limb deformations—that even GPT-4o misses.
- Interpretability: By providing a
<think>block, FingER offers transparency, allowing developers to understand exactly why a video received a specific score.
5. Limitations and Future Work
While powerful, FingER introduces higher computational overhead due to the generation of reasoning chains for multiple entities. Additionally, the framework's effectiveness is partially dependent on the initial LLM's ability to generate comprehensive questions.
Future research may explore using FingER as a reward model within a Reinforcement Learning from Human Feedback (RLHF) loop. This would allow video generation models to learn directly from FingER’s fine-grained feedback, potentially eliminating common artifacts during the training phase itself.
6. Conclusion
FingER represents a significant shift in AI video evaluation, moving from holistic, "black-box" scoring to transparent, entity-level reasoning. By focusing on the specific components of a scene and utilizing advanced RL techniques, FingER provides a more accurate and interpretable metric for the next generation of video synthesis models.
Summary updated