Instruction-guided video editing means changing a clip according to a sentence such as "make it a snowy night." These models are trained almost entirely on synthetic examples, because real pairs of raw footage and the matching edited version are almost never published. The standard recipe edits one frame with an image editor, spreads that edit across the clip with a video generator, and keeps the result if an automated judge approves it. Each training example is therefore a single attempt by a single generator from a single starting frame. Clinical researchers would call that a case report: it shows the edit can happen on this video, but not that the edit is well defined, achievable, or anything more than a lucky accident.
We propose training only on edits that reproduce. For each source video and instruction, we generate several independent edited versions. They start from different frames, use different image editors, and use different random seeds. We keep the example only if these versions agree on what the edit is. The training target is the most representative version, not the one the judge likes best. Our claim is that replication-gated synthetic data should do two things. It should raise instruction compliance without the usual loss of faithfulness to the source. Because no edit is privileged to begin at frame zero, it should also reduce the "first-frame bias" that current benchmarks identify as a universal failure.
Core Claim
The single-case literature makes a sharp distinction between proof of existence and proof of pattern. Tanaka et al. report a 94.0% tumor volume reduction in one patient and state plainly that one case cannot show whether the response is reproducible. Zhou et al. note that a good surgical outcome may reflect the patient's unusual baseline fitness rather than the procedure. Yasuda et al. warn against extrapolating beyond the specific patient profile. These papers supply no method. They supply a principle: an outcome observed once is confounded with the idiosyncrasies of that one instance.
Current synthetic video-editing data violates this principle by construction. InsViE-1M edits the first frame, screens it with GPT-4o, propagates it, and filters with optical flow and GPT-4o. Each kept triplet is one route through one generator. InstructVid2Vid uses Tune-A-Video as its offline expert, and Tune-A-Video is itself a per-video, single-case optimizer. The student inherits that expert's artifacts as a hard ceiling. In both pipelines, an approved example may reflect an easy video, a lucky seed, or a judge error rather than a learnable mapping from instruction to edit.
Our claim is specific and testable. Suppose the generation budget is held fixed, and examples are selected by agreement across independent generation routes rather than by single-sample judge approval. Then a student trained on the result will score higher on instruction compliance in IVEBench at equal or better fidelity. Its gains should concentrate on edits that occur late in the clip, which is where first-frame propagation fails.
Research Program
Data construction. We start from the same pool of source videos and instructions that a standard pipeline would use. For each pair we produce $K$ candidate edited videos, typically $K=4$ to $6$, through deliberately independent routes. The anchor frame on which the image edit is applied is drawn from the first, middle, or final third of the clip. The image editor alternates between two different models; Qwen-Image-Edit, as used in CounterVid, is one option. The propagation seed varies. Propagating from a mid-clip anchor requires a generator that can be conditioned on frames at arbitrary positions. CounterVid shows that Wan2.1 can synthesize video connecting given keyframes, and EditVerse builds its pairs with VACE, so we reuse that class of keyframe-conditioned generator rather than inventing one.
Replication score. For candidates $v_1,\dots,v_K$ of the same pair, we compute
$$R = \frac{2}{K(K-1)} \sum_{i<j} s(v_i, v_j),$$
where $s$ measures agreement between two candidates inside the edited region. The region is localized with Grounded-SAM-2-style masks as in EditVerse. Agreement combines visual feature similarity with a vision-language model's yes/no judgment that both candidates show the same attribute change. We measure agreement only in the edit region on purpose. Outside it, every candidate trivially agrees with the source, so whole-frame similarity would reward edits that do nothing.
A pair is kept if $R$ exceeds a threshold and a single-candidate compliance check also passes. The target is the medoid, meaning the candidate with the highest mean agreement with the others. Thresholds are set per instruction category, for a reason explained below.
Edits with a start time. IVEBench reports that models fail when an edit occurs later in the clip or during an unfolding action. We therefore add instructions with an explicit start time. For these, the frames before the start are copied from the source, and anchors are drawn only from the frames after it. Replication across those post-start anchors is what certifies the example. This produces mid-sequence edits without ever editing frame zero. That directly targets the propagation design that IVEBench blames for first-frame bias.
Training. We train one fixed student architecture on each dataset variant with identical steps, resolution, and hyperparameters. The student is an open video diffusion model fine-tuned for instruction editing, and we choose a model that fits the 832×480 resolution regime that IVEBench reports as typical for open models. The only variable across conditions is how training pairs were selected.
Evaluation that mirrors the cohort logic. Beyond mean scores, we treat each test video as a case. Every model is run with three seeds per test pair. We then report two quantities: a response rate, the fraction of pairs where compliance and fidelity both clear fixed thresholds; and a split of score variance into within-video variance (seed luck) and between-video variance (video difficulty). This tells us whether a gain comes from fewer lucky or unlucky runs or from success on more kinds of video. It also lets us check whether the trade-off between doing the edit and staying faithful to the source, documented in EditBoard and IVEBench, happens within individual videos or only across them.
Baselines
The naive reference point is the backbone's own zero-shot editing behavior, or the backbone fine-tuned on single-route data with no filtering at all. It establishes how much any curation helps.
The strongest existing approach is single-route data with an InsViE-1M-style two-stage judge filter: first-frame image edit, GPT-4o screen, propagation, then optical-flow and GPT-4o screens, as described in InsViE-1M. For fairness, we re-implement this pipeline on our source pool with the same generators. We spend the same total generation budget, so it produces about $K$ times as many single-route candidates as our method has pairs, and it filters them down. InsViE-1M reports a temporal-consistency CLIP score of 0.956 on its own evaluation. On IVEBench, the strongest reported reference is a ceiling rather than a per-model figure: across eight models, "Instruction Compliance rarely exceeding 0.5" and "no model currently achieves a Total Score higher than 0.7" (IVEBench). We found no per-model IVEBench numbers for InsViE-1M or EditVerse on Lacuna, so both must be re-run.
EditVerse is the strongest released system and serves as a reference model on EditVerseBench. Its 51.7% human preference over Runway Aleph sets the bar for what released systems achieve. It is not a matched-compute comparison, because it was trained on 12M multi-task samples.
The key ablation is best-of-$K$. It uses the same $K$ candidates, but the judge picks the single highest-scoring one and nothing is required to agree. This isolates replication as the ingredient. A second ablation, random-anchor, uses a single route but with the anchor drawn uniformly over time. It isolates whether escaping frame zero alone explains any gain on late-onset edits.
Beating the strongest baseline means three things on the IVEBench Short and Long subsets under matched generation budget, averaged over three training seeds. First, IVEBench Instruction Compliance must rise with non-overlapping bootstrap intervals. Second, Video Fidelity must not drop. Third, compliance on the late-onset and action-progression subcategories must improve by more than on first-frame-friendly categories such as style edits.
Why not just…?
Why not just pick the best of $K$ candidates with a stronger judge? This is the most serious threat, and the best-of-$K$ ablation exists to settle it. Our argument is that maximizing a single judge's score selects for whatever the judge over-rewards. An aggressive edit that bleeds into the background can look highly compliant, and IVEBench and EditBoard both report that aggressive models gain compliance at fidelity's expense. Agreement across routes cannot be won by one judge error. If best-of-$K$ matches replication gating on both compliance and fidelity, then our contribution reduces to "generate more and filter," and we will report it that way.
Why not just scale the synthetic data? InsViE-1M already has a million first-frame-propagated triplets. Yet IVEBench's diagnosis is that the first-frame design itself, not dataset size, causes the late-edit failures. Our matched-budget comparison spends the same generation compute on more single-route pairs, so it tests directly whether size or selection matters more. We also include a curve over the size of the single-route set to see whether the gap closes with scale.
Why not just randomize the anchor frame? It is cheaper, and it might remove most of the first-frame bias by itself. The random-anchor ablation tests exactly this. We expect it to help on late-onset edits but to bring in noisier targets, because an edit propagated backward and forward from a single middle frame has no check on whether it holds at the clip's ends. If random-anchor matches us, the replication score is unnecessary for the temporal claim.
Why not just let the model transfer edits from image data, as EditVerse does? EditVerse shows that edit types never seen in its video data, such as weather and material changes, can transfer from image pairs. Our approach is complementary rather than competing. Transfer still needs some video pairs, and EditVerse uses 232K. We expect replication gating to improve whatever video pairs are used. We test this by applying our filter to an EditVerse-style mixed corpus where training compute permits. Inference costs for EditVerse are 118 seconds and 30GB at 360p, which limits how many configurations we can run.
Proposed Experiments and Decisive Checks
The main experiment trains the fixed student under five conditions: unfiltered single-route, judge-filtered single-route, best-of-$K$, random-anchor, and replication-gated. Every condition uses the same generation budget. We evaluate on IVEBench across both subsets and all twelve metrics, on EditVerseBench, and on EditBoard. On EditBoard we break results down by its four difficulty tiers, because the multi-object, multi-attribute tier is where its reported balanced model, Ground-A-Video, separated from the others.
The first decisive check is the per-subcategory compliance gap on IVEBench, comparing late-onset, quantity, and camera-angle edits against style edits. If replication gating helps style edits as much as temporally hard edits, the temporal mechanism we claim is wrong.
The second check targets the clearest failure mode: replication may filter out exactly the hard edits. Independent routes may rarely agree on quantity or camera edits. Then the kept data would be biased toward easy, conservative edits, and the student would become a conservative model of the kind IVEBench describes for InsV2V. We report the fraction of pairs retained per category under the global threshold. We then compare the global threshold with per-category thresholds tuned to the same overall retention. If per-category thresholds do not recover compliance on hard categories, replication is buying fidelity at the cost of coverage, and the claim fails.
The third check is the within-versus-between-video variance split at test time. A real improvement should mainly reduce between-video variance, meaning success across more kinds of video. Reducing only seed variance would mean the student learned to be consistent rather than to edit better.
Finally, we validate the replication score itself. On a held-out set of a few hundred synthetic pairs, humans rate target correctness, and we test whether $R$ predicts those ratings better than the single-candidate GPT-4o score. If it does not, the selection rule has no grounding beyond its downstream effect.
References
- A RESECTED CASE OF ADVANCED ESOPHAGEAL CARCINOMA SHOWING A PROMINENT RESPONSE TO NEOADJUVANT CHEMOTHERAPY WITH 5-FU AND LEUCOVORIN
- Laparoscopic pancreaticoduodenectomy in A-92-older Chinese patient for cancer of head of the pancreas
- Gastrectomy for a patient with early gastric cancer and human immunodeficiency virus (HIV) infection
- InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Construction
- InstructVid2Vid: Controllable Video Editing with Natural Language Instructions
- IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessment
- CounterVid: Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Models
- EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning
- EditBoard: Towards a Comprehensive Evaluation Benchmark for Text-Based Video Editing Models