EVF-Bench: A Comprehensive Benchmark for E-commerce Visual Effects in Video Generation

Run Ling, Ke Cao, Runze He, Jie Gao, Ao Ma, Wei Feng, Zhanjie Zhang, Hao Wang, Jinghe Hu, Guibing GuoarXiv 2026 · arXiv

TL;DR

EVF-Bench is a new, specialized benchmark designed to evaluate how well AI video generators handle e-commerce visual effects (EVF). Alongside a massive dataset of 32,000+ curated triples (EVF-32K), the authors introduce six evaluation dimensions that go beyond simple aesthetics to measure critical factors like product consistency, text stability, and creative rationality.

What Problem Does This Paper Address?

Generating a "cool video" isn't enough for e-commerce. In this domain, a video must balance dynamic visual effects with the rigid requirement of preserving a product's identity and keeping promotional text readable.

General-purpose benchmarks like VBench or EvalCrafter often miss these nuances. For instance, a model might generate a beautiful, high-motion video that accidentally morphs a smartphone into a generic slab or renders a "50% OFF" banner as unreadable gibberish. The authors identify a "task alignment" gap: existing metrics don't account for the coexistence of imaginative effects and reliable product information.

Examples of EVF videos across different product domains, showing the blend of product focus and dynamic effects.

Methods

The researchers propose a six-dimensional evaluation framework. While two are general (Visual Quality and Text Relevance), the other four are tailored specifically for e-commerce:

  1. Product Consistency (ProductScore): Uses SAM2 segmentation and product-name-guided grounding to isolate the product in the reference image. It then evaluates how well the generated video maintains that specific product's appearance, preventing "identity drift."
  2. Text Stability (TSScore): Employs OCR to track promotional text across frames. It measures Content Stability (character error rate over time), Clarity (visual sharpness), and Temporal Coverage (how long the text persists).
  3. Motion Persistence (MotionXScore): Extends standard motion metrics with an "activation ratio." This distinguishes between sustained, meaningful motion and isolated, glitchy bursts.
  4. Creative Rationality (CRScore): Uses GPT-5 as a "multimodal judge" to evaluate if the creative effects are contextually appropriate. For example, "melting cheese" is rational for a pizza ad, but "melting sneakers" might not be.
Fig. 2. Design of the four EVF-specific metrics in EVF-Bench. ProductScore compares the reference product with its generated appearance; TSScore follows rendered text across sampled frames; MotionXScore characterizes motion through optical flow; and CRScore uses the video, generation prompt, and evaluation instruction to judge creative rationality. AesScore and GmeScore provide complementary assessments of visual quality and text relevance and are not shown here.

Datasets

The paper introduces EVF-32K, a dataset of over 32,000 image–text–video triples.

  • Scope: Covers six domains: Food & Beverages, Fashion & Luxury, Home & Appliances, Lifestyle & Leisure, Electronics & Mobility, and Beauty & Health.
  • Curation: Unlike raw web-scraped data, EVF-32K underwent a six-stage pipeline including SKU filtering, image-level quality checks, and manual review to remove "illogical" videos or those with severe subject occlusion.
  • Benchmark: The evaluation component, EVF-Bench, consists of 176 high-quality evaluation cases used to stress-test models.
The product-category taxonomy of EVF-32K, showing the diversity of items covered in the dataset.

Key Insights and Ideas

  • No "One Model to Rule Them All": The evaluation of four commercial (e.g., Kling 2.1, Taobao Xingchen) and four open-source models (e.g., HunyuanVideo, Wan2.2) showed that no single model leads in all six dimensions. There is a clear trade-off between high-motion creativity and strict product/text preservation.
  • Commercial vs. Open-Source Gap: Commercial systems generally outperform unadapted open-source models in product consistency and text stability, likely due to proprietary "safety" or "identity" modules.
  • The Power of Domain Adaptation: By fine-tuning open-source backbones (like Wan2.2) on the EVF-32K dataset using LoRA, the researchers created "EVF-variants" that significantly closed the gap with commercial leaders, even surpassing them in specific metrics like ProductScore.
  • Text is the Hardest Part: Maintaining stable, readable text remains a significant hurdle for most current video diffusion models, often resulting in flickering or "hallucinated" characters.

Key Results

  • Aesthetics & Motion: Kling 2.1 Std achieved the highest scores in Visual Quality (31.76%) and Motion (56.58%).
  • Text Relevance: Taobao Xingchen led in semantic alignment between the prompt and video (59.03%).
  • E-commerce Specifics: The domain-adapted Wan2.2-EVF took the lead in Product Consistency (50.71%) and Creative Rationality (67.23%), proving that task-aligned data is more important than raw model size for these specific requirements.
  • Human Alignment: The authors validated their automated metrics against human preferences, finding high agreement (ranging from 75% to 88%) across the four EVF-specific dimensions.

Limitations and Future Work

  • LLM as Judge: While CRScore uses GPT-5, the authors acknowledge that LLMs still struggle with complex spatial reasoning. Future iterations might move toward more specialized geometric or physical consistency checkers.
  • Commercial Success: The benchmark measures technical and aesthetic quality, but not "advertising effectiveness" (e.g., click-through rates or conversion).
  • Temporal Length: The current benchmark focuses on ~5-second clips. Extending this to longer-form marketing content with multiple scenes remains an open challenge.

This work provides a much-needed "north star" for researchers building AI tools for the multi-billion dollar e-commerce video industry, shifting the focus from "pretty pixels" to "reliable products."

Summary updated

Abstract

E-commerce visual effects (EVF) are dynamic, product-centered visual presentations in e-commerce videos, often accompanied by promotional text. They play an important role in attracting viewer attention and conveying product information. However, general measures of visual quality and text relevance alone do not establish whether generated videos preserve product appearance, maintain readable text, and produce coherent, contextually appropriate effects. To address this issue, we introduce EVF-Bench, a dedicated benchmark spanning six evaluation dimensions: visual quality and text relevance as general dimensions, and motion, product consistency, text stability, and creative rationality as EVF-specific dimensions. To complement the benchmark and support task-aligned research and model adaptation, we further construct EVF-32K, a dataset of over 32000 curated image-text-video triples tailored to e-commerce scenarios. Using this suite, we systematically evaluate representative video generation models and their domain-adapted variants. The results reveal complementary strengths, with no evaluated configuration leading all six dimensions. Together, EVF-Bench and EVF-32K provide a task-oriented evaluation framework and a supporting data resource for e-commerce video generation.
BibTeX
@misc{ling2026evf,
  author = {Run Ling and Ke Cao and Runze He and Jie Gao and Ao Ma and Wei Feng and Zhanjie Zhang and Hao Wang and Jinghe Hu and Guibing Guo},
  title = {{EVF-Bench: A Comprehensive Benchmark for E-commerce Visual Effects in Video Generation}},
  year = {2026},
  eprint = {2611.24116},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24116},
}
Figure