TL;DR
GRNEdit is a highly efficient, lightweight framework that reframes instruction-based video editing as simple "retain-or-flip" decisions over binary latents rather than complex continuous feature regression. By leveraging Generative Refinement Networks (GRNs) and a two-stage alignment process, it achieves state-of-the-art video editing quality while using less than 3% additional conditioning parameters.
What Problem Does This Paper Address?
Most state-of-the-art video editing models operate within the continuous latent spaces of large diffusion or flow-matching backbones. To inject editing instructions and source video context, these systems typically rely on heavyweight conditioning branches (like copied network blocks or adapters) or computationally expensive source token concatenation.
The fundamental challenge is that these continuous models must simultaneously solve two hard problems: where to edit (localization) and what high-dimensional values to generate (synthesis). As generative backbones scale up, the computational cost and optimization burden of these conditioning paths grow prohibitively large. The authors ask a compelling question: Can content synthesis remain entirely with the pretrained backbone, while a lightweight branch models only the editing intent?
Methods
GRNEdit achieves this separation by building on top of Generative Refinement Networks (GRNs), which encode video latents as hierarchical binary codes using Hierarchical Binary Quantization (HBQ). Because the source and target videos share aligned binary coordinates, editing simply requires deciding whether to retain or flip individual bits.
The framework operates in two distinct stages:
- Stage I: Evidence Assimilation and Identity Anchoring: Lightweight source encoders map source bits into continuous "evidence" signals at selected Transformer chunks. Crucially, the authors redefine the classic "null-prompt" dropout used in classifier-free guidance. Instead of dropping the text condition to perform unconditional generation, an empty instruction is explicitly treated as a command to perform no edit, supervised directly via source video reconstruction. This anchors the model's ability to preserve unedited content.
- Stage II: Identity-Referenced Bit-Margin Revision: The framework freezes Stage I and uses the identity path to generate a "source-preserving" reference state in the exact same representation space as the edited state. A lightweight Bit-Margin Router directly compares the edited logits with this preservation reference, using the discrepancy to cleanly revise residual errors and restore fine details.
Datasets
- Training: The models were trained on a task-balanced subset of 0.6M training pairs derived from the public OpenVE-3M dataset. Multimodal Large Language Models (MLLMs) were used to generate source-aware reprompts to enrich the text instructions.
- Evaluation: The framework was evaluated using OpenVE-Bench (evaluating instruction following, source preservation, and temporal quality across 8 editing tasks) and ReCo-Bench (focusing on Add, Replace, Remove, and Style tasks).
Key Insights and Ideas
- The Binary Advantage: Quantizing the video into bits reduces the local search space from an infinite continuum (or a massive discrete codebook) to a simple binary choice. This shifts the conditioning path's job from synthesis to localization-only control.
- Source as "Evidence", Not Features: Instead of treating the source video as a residual target representation to be mathematically combined with the generation, it is treated as coordinate-wise evidence that biases the backbone's native binary decision axis.
- Identity Anchoring: Reinterpreting the null condition as an identity mapping provides a perfect, coordinate-aligned baseline for preservation. This allows Stage II to perform precise logit-level corrections by simply looking at where the edited stream drifted too far from the preservation stream.
Key Results
- State-of-the-Art Efficiency: GRNEdit-2B and GRNEdit-8B achieved overall scores of 4.03 and 4.18 on OpenVE-Bench, respectively.
- Outperforming Larger Models: The 2B parameter model outperformed multiple 14B open-source video editors, while requiring less than 3% additional conditioning parameters (only 37M parameters for the 2B model and 45M for the 8B model).
- Speed: Due to its lightweight conditioning architecture, per-video inference times were remarkably low: 39 seconds for the 2B model and 84 seconds for the 8B model.
- Excellent Localization: GRNEdit performed exceptionally well on localization-sensitive tasks, securing top open-source scores for Background Change and Local Remove tasks.
Limitations and Future Work
The authors note a key structural limitation: Object Addition (the ADD task). Because the binary evidence perspective relies on modifying or retaining existing source structures, it struggles when the target semantics are entirely absent from the source video (e.g., adding a flying seagull to an empty sky). In these scenarios, the lightweight evidence path faces a preservation-generation conflict that it lacks the capacity to fully resolve.
Future work includes allocating richer semantic capacity specifically for object-addition tasks, implementing region-aware gating to relax source preservation in edit zones, and incorporating higher-quality training data that provides stronger physical supervision (shadows, occlusions, and lighting reflections).
Summary updated