TL;DR
RefineCut introduces a new paradigm for video editing by treating it as an explicit, verifiable planning problem rather than an implicit text-generation task. By training a compact 8B open-weight model using a deterministic verifier and rubric-based self-improvement, the authors create a local planner that achieves a 0.924 Video-Editing Score—matching or outperforming massive frontier models like GPT-5.4 and Qwen3-Max in a closed loop.
What Problem Does This Paper Address?
When we think of AI video editing, we often picture pixel generation (like text-to-video models). However, real-world video editing is primarily a highly constrained decision-making process: an editor must look at a pool of footage, select the right clips, trim them down, arrange them in order, add transitions, and align cuts to the beat of a soundtrack—all while hitting a strict target duration.
Current AI video-editing agents handle this "decision layer" by wrapping massive, closed-source frontier models in complex prompt scaffolds. This approach has major drawbacks:
- Implicit Planning: The models guess the timeline implicitly without their decisions being checked against hard constraints.
- High Cost & Latency: Relying on commercial APIs for iterative video adjustments is slow and expensive.
- No Policy Updates: Because the backbone models are frozen, the editing policy cannot be optimized or fine-tuned for the specific task.
The authors address this gap by shifting from implicit prompted planning to explicit verifiable planning, training a compact, local model that can generate and refine structured edit plans.
Methods
RefineCut formalizes video editing as generating a RefinePatch—a structured JSON patch (RFC 6902 style) that modifies a typed timeline state. The framework consists of a two-stage training pipeline:
- Verifier-Replayed Distillation (Stage 1): The authors collect editing trajectories from three frontier models (GPT-5.4, Qwen3-Max, DeepSeek-V4-Pro). Because editing has no single ground truth, teacher suggestions are often noisy or incomplete. Instead of directly imitating the teachers, RefineCut replays every candidate branch through a deterministic verifier. Only the verifier-best branches are kept to serve as targets for Supervised Fine-Tuning (SFT) and preference pairs.
- RefineCut-Evo Self-Improvement (Stage 2): To eliminate reliance on external teachers, the student model generates its own repair candidates. These are scored using a combination of the deterministic verifier and a task-specific editing rubric (covering intent, ledger satisfaction, clip grounding, timeline coherence, pacing, music alignment, and edit economy). The model is then optimized via Direct Preference Optimization (DPO) on high-margin pairs.
At inference time, the planner runs entirely locally in a closed loop with the verifier for up to 3 steps, requiring zero teacher API calls.
Datasets
The authors introduce RefineCut-Bench, a benchmark specifically designed for planning-level video editing. It includes:
- 7,971 real-shot captioned video clips.
- 499 music tracks with beat and energy metadata.
- 3,578 canonical tasks across 3 main families: multi-clip composition, targeted insertion/replacement, and generative placeholder composition (spanning 15 total subtypes).
- Explicit Constraint Ledgers: Machine-checkable requirements detailing target durations, forbidden items, style consistency, and transition rules.
Key Insights and Ideas
- Frontier Teachers are Noisy Guessers: Replaying teacher trajectories through the verifier revealed that frontier models almost never satisfy an entire editing ledger in a single step (hard constraint pass rate $\le$ 0.03). Blindly imitating them causes smaller models to inherit this structural noise.
- Verifiability is a Superpower: Unlike open-ended creative writing, the structural layout of a video edit can be mathematically verified against a ledger. Leveraging this determinism allows execution-based feedback to guide language model optimization effectively.
- Less Data, Better Quality: Training on a dataset three times smaller but strictly filtered by the verifier (3,317 verified examples vs. 9,690 raw teacher examples) yielded massive performance gains, proving that data verification trumps raw data volume in structured planning tasks.
Key Results
- Substantial Performance Gains: On RefineCut-Bench, direct prompting on an untrained Qwen3-8B backbone yields a Video-Editing Score (VES) of 0.594. Raw teacher imitation only nudges this to 0.620. Verifier-replayed SFT drastically boosts the score to 0.858, and the final RefineCut-Evo stage achieves 0.924.
- Beating the Teachers: Operating within the exact same closed-loop testing environment, the local 8B RefineCut-Evo model outperforms GPT-5.4 (0.893) and Qwen3-Max (0.773), while statistically tying with DeepSeek-V4-Pro (0.936).
- Cross-Backbone Transfer: The benefits of verifier-replayed training are not model-specific; applying the same recipe to Llama-3.1-8B and GLM-4-9B consistently eliminated harmful imitation noise and recovered significant performance gains.
Limitations and Future Work
- Planning-Layer Scope: The verifier evaluates structural correctness (durations, clip constraints, pacing rules) but cannot judge artistic taste, emotional impact, or narrative nuance. A plan can be perfectly valid according to the ledger but still feel uninspired to a human viewer.
- Upstream Perception Dependency: The planner does not look at raw video pixels; it relies entirely on text captions and audio metadata. Any errors or omissions from upstream captioning or beat-tracking tools create an immediate bottleneck for the planner's quality.
- Offline Optimization: RefineCut-Evo currently utilizes an offline DPO strategy. Transitioning this framework into a full online reinforcement learning setup remains a task for future exploration.
Summary updated