Abstract
GenProp is a novel generative framework designed to propagate edits from the first frame of a video throughout the entire sequence. By leveraging large-scale image-to-video (I2V) diffusion models and a specialized "selective" encoder, GenProp handles complex structural changes—such as object removal, insertion, and tracking—while maintaining spatial detail and temporal consistency. Unlike traditional methods, it manages physical effects like shadows and reflections without requiring dense per-frame masks or manual motion paths.
Introduction
Video editing remains a significant challenge in computer vision, primarily due to the dual requirements of spatial precision and temporal coherence. Traditional video propagation techniques typically rely on geometric priors like optical flow or depth maps. However, these methods are prone to error accumulation and struggle with occlusions or significant scene transformations.
While generative diffusion models have shown promise in video editing, they are largely restricted to stylistic or appearance-based modifications. Structural edits, such as inserting a new moving object or removing an existing one along with its cast shadow, often result in temporal flickering or the gradual reappearance of deleted content. Furthermore, many existing tools impose a heavy manual burden by requiring users to provide masks for every frame. GenProp addresses these limitations by framing video editing as a unified generative propagation task.
Methodology
GenProp is built upon large-scale I2V models, utilizing both Diffusion Transformers (e.g., Sora) and U-Net architectures (e.g., Stable Video Diffusion). The framework introduces three primary innovations to ensure high-fidelity propagation.
Selective Content Encoder (SCE)
The SCE is a replicated version of the generation model's early layers. Its primary function is to encode the original video sequence to provide context for the regions that should remain unchanged. Crucially, the encoder is "selective": it is trained to ignore regions undergoing edits, allowing the generative model the freedom to synthesize new content or inpaint missing areas without being constrained by the original pixels in those zones.
Mask Prediction Decoder (MPD) and Region-Aware Loss
To improve the model's focus, an auxiliary Mask Prediction Decoder predicts which spatial regions require modification. This helps disentangle edited areas from the static background. Additionally, a Region-Aware (RA) Loss is employed to supervise edited and unedited regions separately, preventing the original content from "bleeding" back into modified areas—a common failure mode in generative inpainting.
Synthetic Data Generation
To train the model without a massive dataset of paired "before and after" videos, the authors developed a synthetic pipeline using instance segmentation datasets. This pipeline simulates three core tasks:
- Copy-and-Paste: Simulates object insertion by placing objects from one video into another.
- Mask-and-Fill: Simulates object removal and inpainting.
- Color Fill: Simulates tracking by filling an object with a solid color, forcing the model to learn to propagate that color (and its associated effects) through the sequence.
Experimental Results
The model was trained on datasets including YouTube-VOS and SAM-V2. Evaluation was conducted on the standard DAVIS dataset and a new "Challenging Test Set" comprising 30 high-complexity videos featuring large shape changes and difficult backgrounds.
Quantitative and Qualitative Performance
GenProp significantly outperformed state-of-the-art (SOTA) methods like InsV2V, AnyV2V, and Pika. On the Challenging Test Set, GenProp achieved a PSNR_m of 32.163 (measuring background consistency), surpassing Pika's 31.329. It also led in text alignment (CLIP-T) and frame consistency (CLIP-I).
Qualitatively, user studies indicated a strong preference for GenProp. In the challenging alignment category, users preferred GenProp over AnyV2V 97.78% of the time. The model demonstrated an impressive ability to generate "physically plausible" motion; for instance, when blueberries are inserted into a bowl of yogurt, they react naturally to the movement of a spoon.
Conclusion
GenProp represents a significant step forward in generative video editing by treating propagation as a generation problem guided by a selective encoder. While the model currently depends on first-frame initialization and faces inference speed challenges typical of diffusion models, its ability to handle complex structural edits with minimal user input sets a new benchmark for the field. Future work will focus on supporting multiple keyframe edits and improving temporal "memory" for longer video sequences.
Summary updated