Plans You Can Check: Verifier-Grounded Learning of an Open-Weight Planner for Executable Video-Editing

Haoyu Wang, Cheng Feng, Liuyang Bian, Ruiyang Huang, Lei Wei, Yafei Wen, Xiaoxin Chen, Xiaoying TangarXiv 2026 · arXiv

TL;DR

RefineCut introduces a new paradigm for video editing by treating it as an explicit, verifiable planning problem rather than an implicit text-generation task. By training a compact 8B open-weight model using a deterministic verifier and rubric-based self-improvement, the authors create a local planner that achieves a 0.924 Video-Editing Score—matching or outperforming massive frontier models like GPT-5.4 and Qwen3-Max in a closed loop.

What Problem Does This Paper Address?

When we think of AI video editing, we often picture pixel generation (like text-to-video models). However, real-world video editing is primarily a highly constrained decision-making process: an editor must look at a pool of footage, select the right clips, trim them down, arrange them in order, add transitions, and align cuts to the beat of a soundtrack—all while hitting a strict target duration.

Current AI video-editing agents handle this "decision layer" by wrapping massive, closed-source frontier models in complex prompt scaffolds. This approach has major drawbacks:

  1. Implicit Planning: The models guess the timeline implicitly without their decisions being checked against hard constraints.
  2. High Cost & Latency: Relying on commercial APIs for iterative video adjustments is slow and expensive.
  3. No Policy Updates: Because the backbone models are frozen, the editing policy cannot be optimized or fine-tuned for the specific task.

The authors address this gap by shifting from implicit prompted planning to explicit verifiable planning, training a compact, local model that can generate and refine structured edit plans.

Figure 1: From implicit to explicit planning. Re- fineCut turns implicit prompted editing into explicit, verifier-trained executable planning.

Methods

RefineCut formalizes video editing as generating a RefinePatch—a structured JSON patch (RFC 6902 style) that modifies a typed timeline state. The framework consists of a two-stage training pipeline:

  1. Verifier-Replayed Distillation (Stage 1): The authors collect editing trajectories from three frontier models (GPT-5.4, Qwen3-Max, DeepSeek-V4-Pro). Because editing has no single ground truth, teacher suggestions are often noisy or incomplete. Instead of directly imitating the teachers, RefineCut replays every candidate branch through a deterministic verifier. Only the verifier-best branches are kept to serve as targets for Supervised Fine-Tuning (SFT) and preference pairs.
  2. RefineCut-Evo Self-Improvement (Stage 2): To eliminate reliance on external teachers, the student model generates its own repair candidates. These are scored using a combination of the deterministic verifier and a task-specific editing rubric (covering intent, ledger satisfaction, clip grounding, timeline coherence, pacing, music alignment, and edit economy). The model is then optimized via Direct Preference Optimization (DPO) on high-margin pairs.

At inference time, the planner runs entirely locally in a closed loop with the verifier for up to 3 steps, requiring zero teacher API calls.

Figure 3: Framework. Noisy teacher trajectories become verified supervision; verifier- and rubric-scored student repairs drive self-improvement and closed-loop planning.

Datasets

The authors introduce RefineCut-Bench, a benchmark specifically designed for planning-level video editing. It includes:

  • 7,971 real-shot captioned video clips.
  • 499 music tracks with beat and energy metadata.
  • 3,578 canonical tasks across 3 main families: multi-clip composition, targeted insertion/replacement, and generative placeholder composition (spanning 15 total subtypes).
  • Explicit Constraint Ledgers: Machine-checkable requirements detailing target durations, forbidden items, style consistency, and transition rules.
Figure 2: RefineCut-Bench overview. The benchmark ties real clip/music metadata, explicit ledger constraints, multi-teacher trajectories, and verifier-based evaluation into one planning-level protocol.

Key Insights and Ideas

  • Frontier Teachers are Noisy Guessers: Replaying teacher trajectories through the verifier revealed that frontier models almost never satisfy an entire editing ledger in a single step (hard constraint pass rate $\le$ 0.03). Blindly imitating them causes smaller models to inherit this structural noise.
  • Verifiability is a Superpower: Unlike open-ended creative writing, the structural layout of a video edit can be mathematically verified against a ledger. Leveraging this determinism allows execution-based feedback to guide language model optimization effectively.
  • Less Data, Better Quality: Training on a dataset three times smaller but strictly filtered by the verifier (3,317 verified examples vs. 9,690 raw teacher examples) yielded massive performance gains, proving that data verification trumps raw data volume in structured planning tasks.

Key Results

  • Substantial Performance Gains: On RefineCut-Bench, direct prompting on an untrained Qwen3-8B backbone yields a Video-Editing Score (VES) of 0.594. Raw teacher imitation only nudges this to 0.620. Verifier-replayed SFT drastically boosts the score to 0.858, and the final RefineCut-Evo stage achieves 0.924.
Training-stage progression
  • Beating the Teachers: Operating within the exact same closed-loop testing environment, the local 8B RefineCut-Evo model outperforms GPT-5.4 (0.893) and Qwen3-Max (0.773), while statistically tying with DeepSeek-V4-Pro (0.936).
  • Cross-Backbone Transfer: The benefits of verifier-replayed training are not model-specific; applying the same recipe to Llama-3.1-8B and GLM-4-9B consistently eliminated harmful imitation noise and recovered significant performance gains.

Limitations and Future Work

  • Planning-Layer Scope: The verifier evaluates structural correctness (durations, clip constraints, pacing rules) but cannot judge artistic taste, emotional impact, or narrative nuance. A plan can be perfectly valid according to the ledger but still feel uninspired to a human viewer.
  • Upstream Perception Dependency: The planner does not look at raw video pixels; it relies entirely on text captions and audio metadata. Any errors or omissions from upstream captioning or beat-tracking tools create an immediate bottleneck for the planner's quality.
  • Offline Optimization: RefineCut-Evo currently utilizes an offline DPO strategy. Transitioning this framework into a full online reinforcement learning setup remains a task for future exploration.

Summary updated

Abstract

Practical video editing is not only pixel generation: an editor must turn a brief, a clip pool, music metadata, and hard constraints into an executable timeline. We study this decision layer as \emph{executable video-editing planning} and introduce RefineCut, which, unlike workflow systems that wrap a prompted frontier model, trains a compact open-weight planner for it. The planner edits a typed timeline through structured patches covering clip selection, trimming, ordering, transitions, and duration and music alignment; a deterministic verifier applies each patch and checks it against an explicit constraint ledger. Because editing has no single ground-truth repair, we do not imitate teachers directly: RefineCut replays every multi-teacher branch through the verifier and keeps verifier-best repairs as supervision. A second stage, RefineCut-Evo, lets the student score its own repairs with the verifier and a task rubric and trains on high-margin preference pairs, so the final $8$B planner runs in a closed verifier loop with no teacher calls at inference. On RefineCut-Bench ($3{,}578$ tasks, $7{,}971$ captioned clips, $499$ music tracks, explicit ledgers), verifier-replayed distillation lifts the planner from $0.620$ to $0.858$ on the protocol-specific Video-Editing Score and RefineCut-Evo reaches $0.924$; the gain transfers to Llama-3.1-8B and GLM-4-9B, and in the same closed loop the $8$B planner matches or exceeds its frontier teachers. Code and RefineCut-Bench are publicly released; see the Data Availability statement.
BibTeX
@misc{wang2026plans,
  author = {Haoyu Wang and Cheng Feng and Liuyang Bian and Ruiyang Huang and Lei Wei and Yafei Wen and Xiaoxin Chen and Xiaoying Tang},
  title = {{Plans You Can Check: Verifier-Grounded Learning of an Open-Weight Planner for Executable Video-Editing}},
  year = {2026},
  eprint = {2608.25622},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.25622},
}
Figure