Generative Video Propagation

Shaoteng Liu, Tianyu Wang, Jui-Hsien Wang, Qing Liu, Zhifei Zhang, Joon-Young Lee, Yijun Li, Bei Yu, Zhe Lin, Soo Ye Kim, Jiaya Jia

Abstract

GenProp is a novel generative framework designed to propagate edits from the first frame of a video throughout the entire sequence. By leveraging large-scale image-to-video (I2V) diffusion models and a specialized "selective" encoder, GenProp handles complex structural changes—such as object removal, insertion, and tracking—while maintaining spatial detail and temporal consistency. Unlike traditional methods, it manages physical effects like shadows and reflections without requiring dense per-frame masks or manual motion paths.

Introduction

Video editing remains a significant challenge in computer vision, primarily due to the dual requirements of spatial precision and temporal coherence. Traditional video propagation techniques typically rely on geometric priors like optical flow or depth maps. However, these methods are prone to error accumulation and struggle with occlusions or significant scene transformations.

While generative diffusion models have shown promise in video editing, they are largely restricted to stylistic or appearance-based modifications. Structural edits, such as inserting a new moving object or removing an existing one along with its cast shadow, often result in temporal flickering or the gradual reappearance of deleted content. Furthermore, many existing tools impose a heavy manual burden by requiring users to provide masks for every frame. GenProp addresses these limitations by framing video editing as a unified generative propagation task.

Methodology

GenProp is built upon large-scale I2V models, utilizing both Diffusion Transformers (e.g., Sora) and U-Net architectures (e.g., Stable Video Diffusion). The framework introduces three primary innovations to ensure high-fidelity propagation.

The GenProp training framework and synthetic data pipeline.

Selective Content Encoder (SCE)

The SCE is a replicated version of the generation model's early layers. Its primary function is to encode the original video sequence to provide context for the regions that should remain unchanged. Crucially, the encoder is "selective": it is trained to ignore regions undergoing edits, allowing the generative model the freedom to synthesize new content or inpaint missing areas without being constrained by the original pixels in those zones.

Mask Prediction Decoder (MPD) and Region-Aware Loss

To improve the model's focus, an auxiliary Mask Prediction Decoder predicts which spatial regions require modification. This helps disentangle edited areas from the static background. Additionally, a Region-Aware (RA) Loss is employed to supervise edited and unedited regions separately, preventing the original content from "bleeding" back into modified areas—a common failure mode in generative inpainting.

Synthetic Data Generation

To train the model without a massive dataset of paired "before and after" videos, the authors developed a synthetic pipeline using instance segmentation datasets. This pipeline simulates three core tasks:

  • Copy-and-Paste: Simulates object insertion by placing objects from one video into another.
  • Mask-and-Fill: Simulates object removal and inpainting.
  • Color Fill: Simulates tracking by filling an object with a solid color, forcing the model to learn to propagate that color (and its associated effects) through the sequence.

Experimental Results

The model was trained on datasets including YouTube-VOS and SAM-V2. Evaluation was conducted on the standard DAVIS dataset and a new "Challenging Test Set" comprising 30 high-complexity videos featuring large shape changes and difficult backgrounds.

Visual examples of GenProp's capabilities across various video editing tasks.

Quantitative and Qualitative Performance

GenProp significantly outperformed state-of-the-art (SOTA) methods like InsV2V, AnyV2V, and Pika. On the Challenging Test Set, GenProp achieved a PSNR_m of 32.163 (measuring background consistency), surpassing Pika's 31.329. It also led in text alignment (CLIP-T) and frame consistency (CLIP-I).

Qualitatively, user studies indicated a strong preference for GenProp. In the challenging alignment category, users preferred GenProp over AnyV2V 97.78% of the time. The model demonstrated an impressive ability to generate "physically plausible" motion; for instance, when blueberries are inserted into a bowl of yogurt, they react naturally to the movement of a spoon.

Conclusion

GenProp represents a significant step forward in generative video editing by treating propagation as a generation problem guided by a selective encoder. While the model currently depends on first-frame initialization and faces inference speed challenges typical of diffusion models, its ability to handle complex structural edits with minimal user input sets a new benchmark for the field. Future work will focus on supporting multiple keyframe edits and improving temporal "memory" for longer video sequences.

Summary updated

Abstract

Large-scale video generation models have the inherent ability to realistically model natural scenes. In this paper, we demonstrate that through a careful design of a generative video propagation framework, various video tasks can be addressed in a unified way by leveraging the generative power of such models. Specifically, our framework, GenProp, encodes the original video with a selective content encoder and propagates the changes made to the first frame using an image-to-video generation model. We propose a data generation scheme to cover multiple video tasks based on instance-level video segmentation datasets. Our model is trained by incorporating a mask prediction decoder head and optimizing a region-aware loss to aid the encoder to preserve the original content while the generation model propagates the modified region. This novel design opens up new possibilities: In editing scenarios, GenProp allows substantial changes to an object's shape; for insertion, the inserted objects can exhibit independent motion; for removal, GenProp effectively removes effects like shadows and reflections from the whole video; for tracking, GenProp is capable of tracking objects and their associated effects together. Experiment results demonstrate the leading performance of our model in various video tasks, and we further provide in-depth analyses of the proposed framework.
BibTeX
@inproceedings{liu2025generative,
  author = {Shaoteng Liu and Tianyu Wang and Jui-Hsien Wang and Qing Liu and Zhifei Zhang and Joon-Young Lee and Yijun Li and Bei Yu and Zhe Lin and Soo Ye Kim and Jiaya Jia},
  title = {{Generative Video Propagation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01650},
}
Figure