Introduction
Evaluating AI-driven video editing is notoriously difficult because standard metrics often ignore human nuance and the relationship between the original and edited footage. While the field of AI-Generated Content (AIGC) is expanding rapidly with new text-to-video editing models, measurement tools have lagged behind. Traditional metrics like CLIP or LPIPS often fail to capture naturalness, prompt adherence, and fidelity to the source video. VE-Bench addresses this gap by introducing the first comprehensive benchmark suite specifically for text-driven video editing, featuring a large-scale human-annotated database (VE-Bench DB) and a novel multi-modal assessment network (VE-Bench QA) that aligns closely with human perception.
Methods
The VE-Bench framework consists of two primary components: a high-quality dataset and a sophisticated evaluation model designed to automate human-like judgment.
VE-Bench DB
To establish a ground truth for quality, researchers curated 169 source videos across diverse categories including human actions, animals, and landscapes. By applying eight state-of-the-art editing models, they generated over 1,100 edited results. These were then rated by 24 human annotators on a scale of 1 to 10, creating a robust database of Mean Opinion Scores (MOS) that serves as the gold standard for video editing quality.
VE-Bench QA
The core technical contribution is a multi-modal network that evaluates three distinct dimensions of video editing:
- Video-Text Alignment: Utilizes a BLIP-based encoder with a temporal adapter to ensure the video content matches the text prompt.
- Source-Target Relationship: Employs spatiotemporal extractors (Uniformer) to measure how well the edit preserves the necessary structure of the original video.
- Visual Quality: Features dedicated branches for Aesthetics and Technical Distortion to detect blurriness or AI-generated artifacts.
Dataset Diversity
The benchmark utilizes a wide array of sources to ensure broad applicability. Source videos were pulled from established datasets like DAVIS, Kinetics-700, Sintel, and Spring, as well as high-end AIGC sources like Sora and Kling. The dataset is balanced across humans, animals, objects, and landscapes, covering both "ego-motion" (first-person) and "exo-motion" (third-person) perspectives.
Key Insights
Analysis of the benchmark revealed several critical trends in current video editing technology:
- The "Removal" Bottleneck: Current AI models struggle significantly with "Removal" tasks (e.g., removing an object from a scene) due to the complexity of background reconstruction.
- Stylization Strengths: Conversely, models excel at style changes (e.g., turning a video into an oil painting), likely due to the extensive artistic data in pre-trained diffusion models.
- Structural Challenges: Structural edits, such as changing the size or pose of an object, remain the most difficult to execute while maintaining temporal consistency.
Results and Performance
VE-Bench QA significantly outperforms existing metrics in its correlation with human judgment. On the VE-Bench DB, the proposed model achieved a Spearman correlation (SROCC) of 0.7415, vastly outperforming traditional zero-shot metrics like CLIP-F (0.2284) and fine-tuned VQA baselines like DOVER (0.6119). Furthermore, the model demonstrated strong generalization capabilities when tested on external datasets like T2VQA-DB, proving it has learned a generalizable sense of video quality rather than merely overfitting to its training data.
Conclusion
While VE-Bench represents a major advancement, challenges remain in quantifying complex motion and "warping" artifacts. Future work may involve integrating Large Language Models (LLMs) to better interpret complex, multi-step editing instructions and expanding the dataset scale to keep pace with the rapidly evolving AIGC landscape.
Summary updated