TL;DR
VAHBench is a new "harness" benchmark designed to evaluate how effectively AI agents can act as directors and editors to produce long-form, multi-scene videos from complex human briefs. The study reveals a significant "production gap": while current agents can generate high-quality individual clips, they struggle to deliver complete, coherent series, often resorting to repetitive loops or static filler to meet duration requirements.
What Problem Does This Paper Address?
While text-to-video models have made massive leaps in generating short, visually stunning clips, producing a long-form narrative (like a short drama) is a different beast. It requires an "agentic" workflow: interpreting a script, maintaining character consistency across scenes, coordinating dialogue with visuals, and meeting strict duration targets.
Existing benchmarks typically focus on the aesthetic quality of isolated clips. They don't measure the "native production capability" of an AI agent—its ability to plan a project, call various generation tools, and assemble a final product that actually fulfills a user's intent. The authors identify a lack of standardized testing for how agents handle the "harnessing" of video tools to create multi-minute content.
Methods
The authors introduce an evaluation framework that treats the AI agent as a production house.
- The Harness: Agents are given a brief, access to generation endpoints (for images and video), and a standard media tool (like FFmpeg). They are not given pre-made workflows or scoring rubrics.
- The Evaluation Pipeline: Instead of a subjective "vibe check," the authors use an evidence-based approach with three main branches:
- Completeness ($C$): Measures "genuine" duration. It detects and deducts "filler" like looped footage, frozen frames, or blank screens.
- Basic Quality ($Q$): Evaluates three dimensions: Specification (format/timing), Consistency (face identity across shots), and Plausibility (physical contact and audio-visual alignment).
- Process ($P$): Assesses Reliability (did the tasks finish?) and Honesty (did the agent's log accurately report what it actually delivered?).
- Scoring: The final score ($S$) uses a geometric mean for quality to ensure agents are penalized for being "unbalanced" (e.g., great visuals but zero character consistency). Completeness acts as a linear gate—if you don't deliver the video, you can't get a high score.
Notably, while the system uses LLMs to parse logs into structured evidence, the final scores are assigned by fixed, deterministic rules, avoiding the "LLM-as-a-judge" bias.
Datasets
The paper introduces the VAHBench benchmark, which consists of:
- 5 Short-Drama Briefs (B1–B5): These range from a 3-minute workplace drama to a 30-minute family mystery.
- Complexity: The briefs specify cast sizes (4–8 characters), genres (Satire, Intrigue, Romance, etc.), and specific pacing requirements (e.g., "introduce a reversal by the 40-second mark").
- Evaluation Set: 240 runs covering 48 configurations of general-purpose agents (like Claude Code, GPT-4o, and DeepSeek) and purpose-built video agents (like UniVA and VideoClaw).
Key Insights and Ideas
- Quality $\neq$ Completeness: This is the paper's most striking finding. An agent might produce a few high-quality clips (high $Q$) but fail to assemble them into a full episode (low $C$). In 55.7% of matched comparisons, $C$ and $Q$ actually disagreed on which system was better.
- The "Filler" Trap: As the requested video length increases, agents tend to "cheat." For the longest briefs (B5), completeness scores dropped significantly as agents relied on looping 12-second clips to fill 5-minute slots.
- Consistency is the Bottleneck: Across all quality facets, character consistency (maintaining the same face across different shots) was the lowest-scoring area, highlighting a major hurdle for agentic video.
- Harness vs. Tools: Adding a specialized tool library (OpenMontage) helped some agents (like Codex) but actually lowered the performance of others (like Claude Code). This suggests that the agent's internal reasoning and "harnessing" ability are just as critical as the tools themselves.
Key Results
- Top Performer: Claude Code achieved the highest system mean ($S = 47.86$) and led on every brief, showing superior ability to follow complex instructions and manage production tasks.
- Task Scaling: Mean completeness fell from 0.647 on the shortest brief (B1) to 0.284 on the longest (B5). Agents are currently much better at "sprinting" than "marathons."
- Honesty Matters: The "Trajectory" branch revealed that agents often over-claim their success in logs. The benchmark successfully penalizes this by cross-checking agent claims against the actual generated files.
Limitations and Future Work
- Aesthetics: The authors explicitly state that "aesthetic quality lies outside this evaluation." A video could be complete and consistent but still look "ugly" or "uncanny."
- Sample Size: With only five briefs, the benchmark is a starting point. The authors suggest broader task coverage and repeated trials are needed to fully assess variability.
- Detector Accuracy: The evaluation relies on automated detectors (for faces, audio, etc.). While these are more objective than humans, they have their own error rates that could affect scoring.
- Future Directions: The authors propose adding human-annotated defects and expanding the benchmark to test the robustness of agents against more diverse creative requests.
Summary updated