Multimodal Judges Flag, Humans Decide: A Review-and-Edit Framework for AI-Generated Product Video Ads

Anoubhav Agarwaal, Sukesh Perla, Karthikeyan Mohanraj, Susmita GhosearXiv 2026 · arXiv

TL;DR

This paper introduces a "Review-and-Edit" framework that combines specialized AI "judges" with human reviewers to scale quality control for AI-generated product video ads. By using multimodal models to flag specific defects across six axes—such as garbled logos or script hallucinations—the system helps humans catch more than twice as many errors while enabling surgical, automated fixes for confirmed issues.

What Problem Does This Paper Address?

Generative AI can now turn an entire e-commerce catalog into thousands of short video ads. However, ensuring these ads are actually usable is a massive bottleneck. Generative models frequently hallucinate: they might add an extra pocket to a pair of pants, garble a brand logo, or claim a suitcase is "unbreakable" when the product listing says no such thing.

Standard video quality metrics (like CLIP similarity or generic aesthetic scores) are too coarse to catch these subtle, product-specific "hallucinations." Furthermore, human reviewers are prone to "attention fade" when checking hundreds of similar ads. The authors address the need for a scalable, precise review process that doesn't just say an ad is "bad," but identifies exactly what is wrong so it can be fixed without re-generating the entire video.

Methods

The authors operationalized the review process into a structured pipeline:

  1. Six Axes of Review: Every ad is evaluated on Product fidelity, Scene quality, Pronunciation, Script accuracy, Sound, and Captions.
  2. Multimodal AI Judges: The team built specialized models for each axis. For example, the Product Judge uses a Vision-Language Model (VLM) to compare video frames against original catalog photos, while the Script Judge checks claims against the text in the product listing.
  3. Human-in-the-Loop: Reviewers use a portal where AI findings are pre-flagged. Reviewers can "thumbs up" to accept a finding, "thumbs down" to reject it, or add new issues the AI missed.
  4. Targeted Editing: Confirmed issues are routed to specific edit tools. If a logo is garbled, a "Repair" tool (using video-to-video models) regenerates just that scene at higher resolution. If a script line is wrong, only that specific narration line is re-voiced or rewritten.
Figure 1. A review-and-edit framework for AI-generated product video ads, built to run at catalog scale. AI judges and the reviewer work on the same six axes, each shown beside the part of the ad it judges (Tab. 1): the judges (blue) flag problems, and the reviewer (orange) answers one question per axis, votes (thumb) on each judge finding (square) and adds what the judges missed (plus). At Decide, the ad is approved, sent to Edit (as here) or rejected (Sec. 3). Edit fixes only what the review found (here, redoing scene 3 and moving its caption), keeps the rest of the ad, and sends the edited ad back to review.

Datasets

The authors introduced and released a new benchmark dataset:

  • 120 generated ads (562 scenes): Includes full human marks and AI judge findings.
  • 120 blind-labeled scenes: Used to evaluate judge performance against human ground truth without the influence of AI flags.
  • Baseline Scores: Comparison data for generic metrics like DINOv2, CLIP, and VideoScore.

The dataset is available at github.com/anoubhav/ad-review-and-edit.

Key Insights and Ideas

  • AI Flags Prevent Human "Attention Fade": One of the most striking findings was that human reviewers become significantly more effective when "anchored" by AI suggestions. Without AI flags, a reviewer's detection rate for product errors dropped from 25% to 12% over a single session. With AI flags, their performance held steady.
  • Recall Over Precision: The authors intentionally tuned their AI judges for high recall. In a production pipeline, a "false flag" only costs a human one click to reject, but a "missed defect" (a bad ad reaching a customer) is highly costly.
  • Generic Metrics are Near-Chance: Popular metrics like CLIP similarity or VideoScore reached AUCs of only 0.48 to 0.57 on product fidelity—barely better than a coin flip. In contrast, their specialized Product Judge reached an AUC of 0.84.
  • Surgical Fixes Save Compute: By mapping specific "axes" to specific "fixes," the system avoids the "lottery" of re-generating a whole ad. If 4 out of 5 scenes are perfect, the system only touches the 5th.
Figure 5. Product marks with judge findings on screen and with judges hidden, same 60 ads. (a) Share of scenes marked product wrong: the round’s majority with judges, the reviewers’ average without. (b) R2, who saw these ads both ways.

Key Results

  • Detection Boost: In a head-to-head test on 60 ads, reviewers with AI judges marked the product "wrong" on 58% of scenes, compared to just 26% for reviewers working without AI assistance.
  • Judge Accuracy: The Product Judge successfully flagged 91% of the scenes that humans eventually marked as defective.
  • Effective Repair: The "Repair" tool, which regenerates scenes at higher resolution based on original prompts, fully corrected garbled product text in 18 out of 26 test cases.
  • Reviewer Agreement: Reviewers showed high agreement (Fleiss’ κ 0.61 to 0.80) on product and scene-quality issues when assisted by judges, though they remained more subjective on script and caption placement.
Figure 4. Each bar splits the items (scenes, lines or captions) any reviewer marked on its axis into those a judge had flagged (blue) and those reviewers added (orange), in the 120-ad round. Black ticks: the share random flags at the judge’s flag rate would cover.

Limitations and Future Work

  • Scope: The study focused on 120 ads for 20 products in the English market. Testing on a wider variety of products and languages is necessary to prove generalizability.
  • Closed Models: The current state-of-the-art judges rely on closed models (like Gemini Flash). While the authors tested open-source alternatives (like Qwen), there is still a performance gap to close.
  • End-to-End Metrics: While the review and edit tools were tested individually, future work is needed to measure the "full loop" efficiency—specifically, what percentage of ads are successfully delivered after a single round of automated editing.
  • Subjectivity in Captions: AI judges still struggle to match human preferences for caption placement and relevance, suggesting these axes require more sophisticated visual-spatial reasoning.

Summary updated

Abstract

Generative pipelines can now turn every product page of an e-commerce catalog into a short video ad, but deciding whether such an ad is usable is not one good-or-bad call. We describe how we operationalized the review and editing of these ads so that both can keep pace with generation: reviewers go through each ad scene by scene and answer six questions, on the product, scene quality, pronunciation, script, sound and captions; judges built from vision-language, speech and language models file each finding under the matching question, and reviewers accept or reject each finding and add what the judges missed. Each confirmed issue then opens an edit tool matched to it. The questions and the routing from a finding to its fix do not depend on the models that generate the ad. Across 120 ads from our pipeline, reviewed with judge findings on screen, reviewers marked the product wrong on 60% of scenes, and script, caption or sound problems on under 10% each. On 60 of these ads, they marked the product wrong on 58% of scenes, far more than the 26% marked by two reviewers who later saw the ads with the judges hidden; one reviewer who saw the ads both ways marked 57% with judge findings and 19% without. On 120 blind-labeled scenes, a product judge that watches the clip with the catalog photos reached AUC 0.84, while video-quality and photo-similarity scores were close to chance. We tune the judges for recall: the product judge catches nearly all defects but also flags many clean scenes, so reviewers decide; together, the judges widen what reviewers mark and reviewers add what the judges miss. Without judge findings, one reviewer's marking faded over a session. We release a review-portal template and our review data (120 ads and 562 scenes with judge findings and human marks, plus 120 blind-labeled scenes with baseline scores) at https://github.com/anoubhav/ad-review-and-edit.
BibTeX
@misc{agarwaal2026multimodal,
  author = {Anoubhav Agarwaal and Sukesh Perla and Karthikeyan Mohanraj and Susmita Ghose},
  title = {{Multimodal Judges Flag, Humans Decide: A Review-and-Edit Framework for AI-Generated Product Video Ads}},
  year = {2026},
  eprint = {2611.13178},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13178},
}
Figure