VIVID: A Visual Harness for Agentic Video Editing

Wenhan Yu, Zhaoxi Zhang, Zhilong Tan, Weikang Li, Lei ShaarXiv 2026 · arXiv

TL;DR

VIVID is a "visual harness" that enables general-purpose coding agents (like Codex or Claude Code) to perform complex video editing tasks without any additional model training. By providing tools for on-demand media inspection, version-linked evidence memory, and iterative output review, VIVID allows these agents to achieve significant performance gains—up to 22.7 percentage points—on professional video post-production benchmarks.

What Problem Does This Paper Address?

While general-purpose coding agents are excellent at reasoning and executing scripts, they are effectively "blind and deaf" when it comes to the nuances of video editing. Professional editing isn't just about running a ffmpeg command; it requires grounding decisions in audiovisual observations, precise temporal localization (knowing exactly which frame a cut should happen), and iterative refinement based on the rendered result.

Existing approaches often rely on specialized, end-to-end trained models that lack the flexibility of general coding agents, or they overwhelm agents with too much raw data. The core challenge VIVID tackles is: How can we give a general-purpose agent the "eyes, ears, and memory" needed for video editing while preserving its native coding capabilities?

Methods

VIVID acts as an interface layer between the agent and the video files. It introduces three core components:

  1. On-demand Audiovisual Inspection: Instead of feeding the whole video into a context window, VIVID provides tools for the agent to "peek" at specific parts. It can request sparse overviews, sample specific frames, crop spatial regions, or analyze audio via waveforms, spectrograms, and transcripts.
  2. Persistent Evidence Memory: Video editing is iterative. When an agent cuts a clip, the timestamps change. VIVID maintains a memory of observations (frames, transcripts) that are explicitly linked to specific media versions and temporal coordinates. This allows the agent to revisit earlier evidence without getting confused by shifting timelines.
  3. Output Inspection and Revision: VIVID supports a "render-inspect-revise" workflow. After an edit, the agent can use the harness to verify the output for technical errors (like aspect ratio distortion) or creative issues (like poor framing) and then go back to the code to fix them.

Crucially, VIVID is model-agnostic. It works by extending the toolset (the "harness") of existing agents rather than retraining the underlying LLM.

Datasets

The authors evaluated VIVID using AgenticVBench, a comprehensive benchmark designed for agentic video editing. It consists of 100 tasks divided into four categories:

  • Assembly: Selecting and combining the correct shots from a pool of candidates.
  • Repair: Fixing timing errors or audiovisual defects.
  • Sequencing: Restoring the correct chronological order of shuffled clips.
  • Repurpose: Transforming long-form footage into new formats (e.g., a highlight reel) based on specific instructions.

Key Insights and Ideas

  • Active Observation vs. Passive Context: The "aha moment" here is that agents don't need to see the whole video at once. By allowing the agent to choose what to inspect based on its current reasoning step, VIVID saves tokens and reduces noise.
  • Traceability is Essential: In video editing, the "truth" changes as you edit. By linking every observation to a specific file version and timestamp, VIVID prevents the agent from making decisions based on outdated information.
  • The Interface is the Bottleneck: The fact that general-purpose models like Claude and Codex saw massive improvements just by changing their tools suggests that the limitation in AI video editing isn't necessarily the models' reasoning ability, but how they interact with media.
  • Iterative Editing: Human editors rarely get it right on the first pass. VIVID proves that giving agents a feedback loop (inspecting their own rendered output) is critical for high-quality results.

Key Results

VIVID was tested across three major agent configurations, showing consistent and substantial improvements over "native" (un-harnessed) baselines:

  • Codex + VIVID: Achieved 58.6% average score (a +20.2 point gain over native Codex).
  • Claude Code + VIVID: Achieved 55.1% average score (a +22.7 point gain).
  • OpenCode + VIVID: Achieved 37.1% average score (a +9.7 point gain).

Ablation studies showed that on-demand inspection was the most critical component for "Assembly" tasks, while evidence memory was the biggest driver for "Repair" tasks. In a case study involving a speech highlight reel, VIVID enabled the agent to identify and fix aspect ratio distortions and missing captions that the native agent completely ignored.

Limitations and Future Work

  • Specialized Processing: While VIVID helps with selection and cutting, the authors noted that "Repair" tasks (like denoising or color correction) still lag behind. Future work could integrate more sophisticated specialized media processing tools.
  • Token Overhead: Although more efficient than full-video processing, VIVID still increases token consumption (between 10% and 53% depending on the agent). Optimizing the "visual vocabulary" sent to the agent remains a challenge.
  • Complex Narrative Reasoning: While the agents improved at sequencing, deep narrative "storytelling" (understanding the emotional arc of a scene) remains a high bar that requires even more sophisticated audiovisual grounding.

Summary updated

Abstract

Video editing requires agents to make editing decisions grounded in audiovisual observations and precise temporal localization, and to evaluate their effects in rendered outputs. General-purpose coding agents possess strong comprehension, reasoning, and code execution capabilities, while media interaction and evidence management mechanisms tailored to video tasks can further unlock their editing potential. We introduce VIVID, a visual harness that supports multiple native coding agents in video post-production without additional model training. VIVID provides tools for on-demand video inspection, audio analysis, and transcription, and preserves observations alongside their corresponding media versions and temporal coordinates. Agents can actively retrieve these observations as they reason, perform edits using their native coding and shell tools, and inspect rendered outputs to guide subsequent revisions. On AgenticVBench, comprising 100 tasks across Assembly, Repair, Sequencing, and Repurpose, Codex with VIVID achieves an average score of 58.6%, an improvement of 20.2 percentage points over native Codex. The same harness also integrates with Claude Code and OpenCode, achieving average scores of 55.1% and 37.1%, respectively, both substantially exceeding their native baselines. These results highlight VIVID's potential as a reusable visual harness for extending the capabilities of general-purpose coding agents to video editing.
BibTeX
@misc{yu2026vivid,
  author = {Wenhan Yu and Zhaoxi Zhang and Zhilong Tan and Weikang Li and Lei Sha},
  title = {{VIVID: A Visual Harness for Agentic Video Editing}},
  year = {2026},
  eprint = {2611.20688},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20688},
}
Figure