TL;DR
EVF-Bench is a new, specialized benchmark designed to evaluate how well AI video generators handle e-commerce visual effects (EVF). Alongside a massive dataset of 32,000+ curated triples (EVF-32K), the authors introduce six evaluation dimensions that go beyond simple aesthetics to measure critical factors like product consistency, text stability, and creative rationality.
What Problem Does This Paper Address?
Generating a "cool video" isn't enough for e-commerce. In this domain, a video must balance dynamic visual effects with the rigid requirement of preserving a product's identity and keeping promotional text readable.
General-purpose benchmarks like VBench or EvalCrafter often miss these nuances. For instance, a model might generate a beautiful, high-motion video that accidentally morphs a smartphone into a generic slab or renders a "50% OFF" banner as unreadable gibberish. The authors identify a "task alignment" gap: existing metrics don't account for the coexistence of imaginative effects and reliable product information.
Methods
The researchers propose a six-dimensional evaluation framework. While two are general (Visual Quality and Text Relevance), the other four are tailored specifically for e-commerce:
- Product Consistency (ProductScore): Uses SAM2 segmentation and product-name-guided grounding to isolate the product in the reference image. It then evaluates how well the generated video maintains that specific product's appearance, preventing "identity drift."
- Text Stability (TSScore): Employs OCR to track promotional text across frames. It measures Content Stability (character error rate over time), Clarity (visual sharpness), and Temporal Coverage (how long the text persists).
- Motion Persistence (MotionXScore): Extends standard motion metrics with an "activation ratio." This distinguishes between sustained, meaningful motion and isolated, glitchy bursts.
- Creative Rationality (CRScore): Uses GPT-5 as a "multimodal judge" to evaluate if the creative effects are contextually appropriate. For example, "melting cheese" is rational for a pizza ad, but "melting sneakers" might not be.
Datasets
The paper introduces EVF-32K, a dataset of over 32,000 image–text–video triples.
- Scope: Covers six domains: Food & Beverages, Fashion & Luxury, Home & Appliances, Lifestyle & Leisure, Electronics & Mobility, and Beauty & Health.
- Curation: Unlike raw web-scraped data, EVF-32K underwent a six-stage pipeline including SKU filtering, image-level quality checks, and manual review to remove "illogical" videos or those with severe subject occlusion.
- Benchmark: The evaluation component, EVF-Bench, consists of 176 high-quality evaluation cases used to stress-test models.
Key Insights and Ideas
- No "One Model to Rule Them All": The evaluation of four commercial (e.g., Kling 2.1, Taobao Xingchen) and four open-source models (e.g., HunyuanVideo, Wan2.2) showed that no single model leads in all six dimensions. There is a clear trade-off between high-motion creativity and strict product/text preservation.
- Commercial vs. Open-Source Gap: Commercial systems generally outperform unadapted open-source models in product consistency and text stability, likely due to proprietary "safety" or "identity" modules.
- The Power of Domain Adaptation: By fine-tuning open-source backbones (like Wan2.2) on the EVF-32K dataset using LoRA, the researchers created "EVF-variants" that significantly closed the gap with commercial leaders, even surpassing them in specific metrics like ProductScore.
- Text is the Hardest Part: Maintaining stable, readable text remains a significant hurdle for most current video diffusion models, often resulting in flickering or "hallucinated" characters.
Key Results
- Aesthetics & Motion: Kling 2.1 Std achieved the highest scores in Visual Quality (31.76%) and Motion (56.58%).
- Text Relevance: Taobao Xingchen led in semantic alignment between the prompt and video (59.03%).
- E-commerce Specifics: The domain-adapted Wan2.2-EVF took the lead in Product Consistency (50.71%) and Creative Rationality (67.23%), proving that task-aligned data is more important than raw model size for these specific requirements.
- Human Alignment: The authors validated their automated metrics against human preferences, finding high agreement (ranging from 75% to 88%) across the four EVF-specific dimensions.
Limitations and Future Work
- LLM as Judge: While CRScore uses GPT-5, the authors acknowledge that LLMs still struggle with complex spatial reasoning. Future iterations might move toward more specialized geometric or physical consistency checkers.
- Commercial Success: The benchmark measures technical and aesthetic quality, but not "advertising effectiveness" (e.g., click-through rates or conversion).
- Temporal Length: The current benchmark focuses on ~5-second clips. Extending this to longer-form marketing content with multiple scenes remains an open challenge.
This work provides a much-needed "north star" for researchers building AI tools for the multi-billion dollar e-commerce video industry, shifting the focus from "pretty pixels" to "reliable products."
Summary updated