TL;DR
This paper introduces a "Review-and-Edit" framework that combines specialized AI "judges" with human reviewers to scale quality control for AI-generated product video ads. By using multimodal models to flag specific defects across six axes—such as garbled logos or script hallucinations—the system helps humans catch more than twice as many errors while enabling surgical, automated fixes for confirmed issues.
What Problem Does This Paper Address?
Generative AI can now turn an entire e-commerce catalog into thousands of short video ads. However, ensuring these ads are actually usable is a massive bottleneck. Generative models frequently hallucinate: they might add an extra pocket to a pair of pants, garble a brand logo, or claim a suitcase is "unbreakable" when the product listing says no such thing.
Standard video quality metrics (like CLIP similarity or generic aesthetic scores) are too coarse to catch these subtle, product-specific "hallucinations." Furthermore, human reviewers are prone to "attention fade" when checking hundreds of similar ads. The authors address the need for a scalable, precise review process that doesn't just say an ad is "bad," but identifies exactly what is wrong so it can be fixed without re-generating the entire video.
Methods
The authors operationalized the review process into a structured pipeline:
- Six Axes of Review: Every ad is evaluated on Product fidelity, Scene quality, Pronunciation, Script accuracy, Sound, and Captions.
- Multimodal AI Judges: The team built specialized models for each axis. For example, the Product Judge uses a Vision-Language Model (VLM) to compare video frames against original catalog photos, while the Script Judge checks claims against the text in the product listing.
- Human-in-the-Loop: Reviewers use a portal where AI findings are pre-flagged. Reviewers can "thumbs up" to accept a finding, "thumbs down" to reject it, or add new issues the AI missed.
- Targeted Editing: Confirmed issues are routed to specific edit tools. If a logo is garbled, a "Repair" tool (using video-to-video models) regenerates just that scene at higher resolution. If a script line is wrong, only that specific narration line is re-voiced or rewritten.
Datasets
The authors introduced and released a new benchmark dataset:
- 120 generated ads (562 scenes): Includes full human marks and AI judge findings.
- 120 blind-labeled scenes: Used to evaluate judge performance against human ground truth without the influence of AI flags.
- Baseline Scores: Comparison data for generic metrics like DINOv2, CLIP, and VideoScore.
The dataset is available at github.com/anoubhav/ad-review-and-edit.
Key Insights and Ideas
- AI Flags Prevent Human "Attention Fade": One of the most striking findings was that human reviewers become significantly more effective when "anchored" by AI suggestions. Without AI flags, a reviewer's detection rate for product errors dropped from 25% to 12% over a single session. With AI flags, their performance held steady.
- Recall Over Precision: The authors intentionally tuned their AI judges for high recall. In a production pipeline, a "false flag" only costs a human one click to reject, but a "missed defect" (a bad ad reaching a customer) is highly costly.
- Generic Metrics are Near-Chance: Popular metrics like CLIP similarity or VideoScore reached AUCs of only 0.48 to 0.57 on product fidelity—barely better than a coin flip. In contrast, their specialized Product Judge reached an AUC of 0.84.
- Surgical Fixes Save Compute: By mapping specific "axes" to specific "fixes," the system avoids the "lottery" of re-generating a whole ad. If 4 out of 5 scenes are perfect, the system only touches the 5th.
Key Results
- Detection Boost: In a head-to-head test on 60 ads, reviewers with AI judges marked the product "wrong" on 58% of scenes, compared to just 26% for reviewers working without AI assistance.
- Judge Accuracy: The Product Judge successfully flagged 91% of the scenes that humans eventually marked as defective.
- Effective Repair: The "Repair" tool, which regenerates scenes at higher resolution based on original prompts, fully corrected garbled product text in 18 out of 26 test cases.
- Reviewer Agreement: Reviewers showed high agreement (Fleiss’ κ 0.61 to 0.80) on product and scene-quality issues when assisted by judges, though they remained more subjective on script and caption placement.
Limitations and Future Work
- Scope: The study focused on 120 ads for 20 products in the English market. Testing on a wider variety of products and languages is necessary to prove generalizability.
- Closed Models: The current state-of-the-art judges rely on closed models (like Gemini Flash). While the authors tested open-source alternatives (like Qwen), there is still a performance gap to close.
- End-to-End Metrics: While the review and edit tools were tested individually, future work is needed to measure the "full loop" efficiency—specifically, what percentage of ads are successfully delivered after a single round of automated editing.
- Subjectivity in Captions: AI judges still struggle to match human preferences for caption placement and relevance, suggesting these axes require more sophisticated visual-spatial reasoning.
Summary updated