Generative Video Propagation

Shaoteng Liu, Tianyu Wang, Jui-Hsien Wang, Qing Liu, Zhifei Zhang, Joon-Young Lee, Yijun Li, Bei Yu, Zhe Lin, Soo Ye Kim, Jiaya Jia

Abstract

GenProp (Generative Video Propagation) is a novel framework designed to seamlessly propagate first-frame edits—such as object removal, insertion, or background replacement—throughout an entire video sequence. By combining a selective content encoder with the generative power of image-to-video (I2V) models, GenProp handles complex physical effects like shadows, reflections, and independent motion without requiring per-frame masks or traditional optical flow. Our approach treats video propagation as a generation task, leveraging the "world-modeling" capabilities of I2V models to ensure temporal consistency and physical realism. Experimental results show that GenProp significantly outperforms state-of-the-art baselines in both quantitative metrics and human preference studies.

Introduction

Video editing typically falls into two categories: propagation-based and generation-based. Traditional propagation relies on intermediate representations like optical flow or point tracking, which are often brittle and accumulate errors over time, especially during significant shape changes or complex occlusions. Conversely, generative models can produce realistic motion but often struggle to maintain consistency with the unedited parts of the original footage.

GenProp bridges this gap by leveraging the creative capacity of generative models to animate new content while strictly preserving the original scene's identity. It treats propagation as a generation task where the model uses its internal "knowledge" of physics to make edits appear natural within the existing environment.

Overview of GenProp's editing capabilities across various tasks, including object removal, background replacement, and object insertion.

Methodology

GenProp is built upon a frozen Image-to-Video (I2V) backbone, utilizing either DiT or U-Net architectures. The framework introduces three key technical innovations:

  1. Selective Content Encoder (SCE): A trainable "sidecar" encoder that processes the original video to identify regions that should remain untouched, fusing these features into the main generation model via an injection layer.
  2. Mask Prediction Decoder (MPD): An auxiliary head trained to predict edited regions, helping the model disentangle new content from the original scene and focus generative efforts only where changes occurred.
  3. Region-Aware (RA) Loss: A specialized loss function that includes a "gradient loss" to prevent "ghosting" by penalizing the encoder if it attempts to reintroduce information from the original (removed) objects into edited areas.
The GenProp training framework and synthetic data pipeline.

To train the model, the authors developed a synthetic data pipeline using YouTube-VOS and SAM-V2, employing strategies like copy-and-paste for insertion, mask-and-fill for removal simulation, and color-filling for robust tracking tasks.

Results and Discussion

GenProp demonstrates superior performance on the TGVE (DAVIS) dataset and a curated "Challenging Test Set" of high-resolution videos featuring complex structural changes. Quantitatively, GenProp achieved a PSNRm of 32.163 on the challenging set, significantly higher than competitors like AnyV2V, ReVideo, and Pika, which typically scored between 28 and 29. In human preference studies with 121 participants, GenProp was preferred 84% to 97% of the time for alignment and quality.

Key Insights

  • Generative Physics: Instead of simple pixel warping, GenProp acts as a physics engine. For example, an inserted object will fall or move naturally based on the scene's context.
  • Automatic Effect Handling: Unlike traditional inpainters that leave "shadow orphans," GenProp successfully removes secondary effects like shadows and reflections because the generative model understands they should not exist without the source object.
  • Zero-Shot Tracking: By training on color-filled synthetic data, the model learns to perform high-quality instance tracking, propagating masks more robustly than some dedicated tracking models.

Conclusion and Future Work

GenProp offers a robust solution for generative video propagation, effectively balancing creative generation with structural preservation. While it excels at visual edits, a current limitation is "causal event removal"—the ability to undo secondary physical consequences of an object's actions, such as removing the splash created by a person jumping into a pool. Future work aims to extend the model to support multi-keyframe edits and address these complex causal relationships.

Summary updated

BibTeX
@inproceedings{liu2025generative,
  author = {Shaoteng Liu and Tianyu Wang and Jui-Hsien Wang and Qing Liu and Zhifei Zhang and Joon-Young Lee and Yijun Li and Bei Yu and Zhe Lin and Soo Ye Kim and Jiaya Jia},
  title = {{Generative Video Propagation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01650},
}
Figure