E-Bench: Subjective-Aligned Benchmark Suite for Text-Driven Video Editing Quality Assessment

Shangkun Sun, Xiaoyu Liang, Songlin Fan, Wenxu Gao, Wei Gao

Introduction

Evaluating AI-driven video editing is notoriously difficult because standard metrics often ignore human nuance and the relationship between the original and edited footage. While the field of AI-Generated Content (AIGC) is expanding rapidly with new text-to-video editing models, measurement tools have lagged behind. Traditional metrics like CLIP or LPIPS often fail to capture naturalness, prompt adherence, and fidelity to the source video. VE-Bench addresses this gap by introducing the first comprehensive benchmark suite specifically for text-driven video editing, featuring a large-scale human-annotated database (VE-Bench DB) and a novel multi-modal assessment network (VE-Bench QA) that aligns closely with human perception.

Methods

The VE-Bench framework consists of two primary components: a high-quality dataset and a sophisticated evaluation model designed to automate human-like judgment.

VE-Bench DB

To establish a ground truth for quality, researchers curated 169 source videos across diverse categories including human actions, animals, and landscapes. By applying eight state-of-the-art editing models, they generated over 1,100 edited results. These were then rated by 24 human annotators on a scale of 1 to 10, creating a robust database of Mean Opinion Scores (MOS) that serves as the gold standard for video editing quality.

VE-Bench QA

The core technical contribution is a multi-modal network that evaluates three distinct dimensions of video editing:

  • Video-Text Alignment: Utilizes a BLIP-based encoder with a temporal adapter to ensure the video content matches the text prompt.
  • Source-Target Relationship: Employs spatiotemporal extractors (Uniformer) to measure how well the edit preserves the necessary structure of the original video.
  • Visual Quality: Features dedicated branches for Aesthetics and Technical Distortion to detect blurriness or AI-generated artifacts.
Overview of the VE-Bench framework, including database construction and the multi-modal quality assessment pipeline.

Dataset Diversity

The benchmark utilizes a wide array of sources to ensure broad applicability. Source videos were pulled from established datasets like DAVIS, Kinetics-700, Sintel, and Spring, as well as high-end AIGC sources like Sora and Kling. The dataset is balanced across humans, animals, objects, and landscapes, covering both "ego-motion" (first-person) and "exo-motion" (third-person) perspectives.

Comprehensive breakdown of the VE-Bench dataset, illustrating the distribution of sources, video types, motion patterns, and content categories.

Key Insights

Analysis of the benchmark revealed several critical trends in current video editing technology:

  • The "Removal" Bottleneck: Current AI models struggle significantly with "Removal" tasks (e.g., removing an object from a scene) due to the complexity of background reconstruction.
  • Stylization Strengths: Conversely, models excel at style changes (e.g., turning a video into an oil painting), likely due to the extensive artistic data in pre-trained diffusion models.
  • Structural Challenges: Structural edits, such as changing the size or pose of an object, remain the most difficult to execute while maintaining temporal consistency.

Results and Performance

VE-Bench QA significantly outperforms existing metrics in its correlation with human judgment. On the VE-Bench DB, the proposed model achieved a Spearman correlation (SROCC) of 0.7415, vastly outperforming traditional zero-shot metrics like CLIP-F (0.2284) and fine-tuned VQA baselines like DOVER (0.6119). Furthermore, the model demonstrated strong generalization capabilities when tested on external datasets like T2VQA-DB, proving it has learned a generalizable sense of video quality rather than merely overfitting to its training data.

Conclusion

While VE-Bench represents a major advancement, challenges remain in quantifying complex motion and "warping" artifacts. Future work may involve integrating Large Language Models (LLMs) to better interpret complex, multi-step editing instructions and expanding the dataset scale to keep pace with the rapidly evolving AIGC landscape.

Summary updated

Abstract

Text-driven video editing has recently experienced rapid development. Despite this, evaluating edited videos remains a considerable challenge. Current metrics tend to fail to align with human perceptions, and effective quantitative metrics for video editing are still notably absent. To address this, we introduce E-Bench, a benchmark suite tailored to the assessment of text-driven video editing. This suite includes E-Bench DB, a video quality assessment (VQA) database for video editing. E-Bench DB encompasses a diverse set of source videos featuring various motions and subjects, along with multiple distinct editing prompts, editing results from 8 different models, and the corresponding Mean Opinion Scores (MOS) from 24 human annotators. Based on E-Bench DB, we further propose E-Bench QA, a quantitative human-aligned measurement for the text-driven video editing task. In addition to the aesthetic, distortion, and other visual quality indicators that traditional VQA methods emphasize, E-Bench QA focuses on the text-video alignment and the relevance modeling between source and edited videos. It proposes a new assessment network for video editing that attains superior performance in alignment with human preferences. To the best of our knowledge, E-Bench introduces the first quality assessment dataset for video editing and an effective subjective-aligned quantitative metric for this domain. All data and code will be publicly available at https://github.com/littlespray/E-Bench.
BibTeX
@misc{sun2024e,
  author = {Shangkun Sun and Xiaoyu Liang and Songlin Fan and Wenxu Gao and Wei Gao},
  title = {{E-Bench: Subjective-Aligned Benchmark Suite for Text-Driven Video Editing Quality Assessment}},
  year = {2024},
  eprint = {2408.11481},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.11481},
}
Figure