RoughCut-Bench: A Benchmark for Automatic Video Rough-Cut with Large Vision-Language Models

Ruisi Zhao, You Li, Sihan Chen, Aosong Cheng, Chixiang Ma, Sen Pei, Zhen Yang, Hehe Fan, Dongliang He, Yi YangarXiv 2026 · arXiv

TL;DR

RoughCut-Bench is a new, comprehensive benchmark designed to evaluate how well Large Vision-Language Models (LVLMs) can handle "rough-cutting"—the foundational stage of video editing where raw footage is selected and sequenced. By introducing specialized agentic workflows and a human-aligned evaluator, the researchers found that while top-tier models like Gemini 3.0 Pro and GPT-5.5 show promise, they still struggle to match the nuanced decision-making of professional editors.

What Problem Does This Paper Address?

In the world of content creation, the "rough-cut" is where the magic (and the grunt work) begins. It involves sifting through hours of raw footage to find the best moments (highlights), deciding their order to tell a story, and cleaning up speech by removing "ums," "ahs," and off-topic tangents.

While LVLMs have become surprisingly good at describing single videos or answering questions about them, they haven't been systematically tested on this compositional creation task. Most existing benchmarks focus on passive understanding rather than active organization. There is a significant gap in both how we evaluate these models for editing and the workflows needed to make them produce actual, executable edit plans.

Methods

The authors propose a two-pronged approach to bridge this gap:

  1. Agentic Workflows: To manage the massive amount of data in multi-video projects, they developed a "decoupled" workflow.
  2. Video-Understanding Agent: Processes each raw video independently to identify candidate highlights, providing reasons and visual descriptions for each.
  3. Cut-Planning Agent: Takes these summaries and descriptions to perform the high-level reasoning—selecting the final clips and determining their order based on a "storyline" or specific user requirements (e.g., "make it 30 seconds long and start with the cooking part").
  4. Transcript-Cleanup Agent: Specifically for speech-driven editing, this agent analyzes ASR (Automatic Speech Recognition) transcripts to remove fillers and redundant expressions.
  5. Human-Aligned Agentic Evaluator: Since "good editing" is subjective, the team built an evaluator (using Gemini 3.0 Pro) that was iteratively refined through four rounds of feedback from professional editors. This ensures the AI's "quality score" actually correlates with what a human pro would think.
Dataset statistics showing the distribution of visual scenarios, video durations, and speech cleanup labels.

Datasets

The paper introduces RoughCut-Bench, which is split into two practical settings:

  • Visually Driven Setting: 101 cases featuring 831 raw videos and nearly 1,500 professional highlight annotations. It covers six real-world scenarios like outdoor vlogs, food tutorials, and daily life.
  • Speech-Driven Setting: 84 cases derived from talking-head videos, focusing on transcript refinement with annotated spans for filler removal and content trimming.

Key Insights and Ideas

  • Rough-cutting is a "Compositional" Task: It’s not just about seeing; it’s about choosing and arranging. A model might be great at finding a "cool shot" but terrible at realizing it belongs at the end of the video, not the beginning.
  • Capability Dissociation: Interestingly, a model’s performance in one area (like removing fillers) doesn't predict its performance in another (like trimming off-topic content). This suggests that "editing" isn't one single skill for an AI, but a collection of distinct reasoning capabilities.
  • The Power of Context: Ablation studies showed that "joint understanding"—letting the model see summaries of all videos at once—improved the quality of highlight selection. Knowing what else is available helps the AI judge if a specific moment is truly a "highlight."
  • Reasoning Matters: Using Chain-of-Thought (CoT) prompting significantly improved the model's ability to trim low-value content, though it was less critical for simple tasks like filler removal.

Key Results

  • Top Performers: Gemini 3.0 Pro and GPT-5.5 emerged as the leaders. Gemini 3.0 Pro achieved the best F1 score (0.526) for highlight selection on general cases.
  • The "User Requirement" Challenge: Models were surprisingly better at ordering videos when given specific user instructions (Kendall’s $\tau$ increased from ~0.56 to ~0.70), likely because the instructions provided a clearer narrative anchor.
  • Evaluator Alignment: After four rounds of refinement, the agentic evaluator reached a 0.98 Pearson correlation with human professional scores, making it a highly reliable tool for automated assessment.
  • The Gap: Despite these successes, the best models still only match professional highlight selections about half the time (F1 scores around 0.5), indicating we are still in the early days of fully automated AI editing.

Limitations and Future Work

The authors acknowledge several areas for growth:

  • Missing Cues: The speech-driven setting currently relies only on text transcripts, ignoring vital audio-visual cues like prosody (tone of voice), emotion, or speaker body language.
  • Separated Modalities: Currently, visual and speech editing are handled as separate tasks. Real-world editing requires a unified "omni-modal" approach where the AI balances what is being said with what is being shown.
  • Long-Context Bottlenecks: While agentic workflows help, the "holy grail" is a model that can natively reason over hours of raw footage in a single pass without losing track of the narrative.

RoughCut-Bench sets a high bar for the next generation of creative AI, moving us closer to a world where your favorite vlog might just have been "rough-cut" by a machine.

Summary updated

Abstract

Video rough-cut is a core stage in media production workflows, organizing raw media materials into an initial video cut. Despite recent advances in large vision-language models (LVLMs) for video-related tasks, automatic rough-cut remains underexplored, without a comprehensive benchmark for evaluation or mature workflows for producing executable cut plans. To address the evaluation gap, we introduce RoughCut-Bench, which covers two practical settings: (1) a visually driven setting over video cases, covering highlight detection and selection, source-video ordering, and adherence to user requirements when provided; and (2) a speech-driven setting over automatic speech recognition (ASR) transcripts, covering filler removal, expression deduplication, and low-value content trimming. Across these settings, the benchmark contains 101 visually driven cases, including 831 raw videos with 1,476 highlight annotations. It also includes 84 speech-driven cases, each consisting of an ASR transcript with annotated cleanup spans. The evaluation compares cut plans with these annotations in both settings and additionally uses a human-aligned agentic evaluator to assess rendered-cut quality in the visual setting. To apply LVLMs to automatic rough-cut, we further propose agentic workflows that guide them to produce structured cut plans from input materials. Experiments show that current LVLMs remain limited in producing expert-aligned cut plans. Our benchmark, code, and prompts are released at https://whalesong-zrs.github.io/RoughCut-Bench-Project-Page.
BibTeX
@misc{zhao2026roughcut,
  author = {Ruisi Zhao and You Li and Sihan Chen and Aosong Cheng and Chixiang Ma and Sen Pei and Zhen Yang and Hehe Fan and Dongliang He and Yi Yang},
  title = {{RoughCut-Bench: A Benchmark for Automatic Video Rough-Cut with Large Vision-Language Models}},
  year = {2026},
  eprint = {2611.24098},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24098},
}
Figure