UniVOE: A Removal-Driven Framework for Interaction-Aware Universal Video Object Editing

Shuzhou Yang, Lanqing Guo, Lingen Li, Weiqi Li, Zhao Zhong, Liefeng Bo, Jian ZhangarXiv 2026 · arXiv

TL;DR

UniVOE is a novel framework that solves the "data scarcity" problem in video object editing by using a high-quality object remover to generate a massive synthetic-but-realistic training dataset for an object injector. By combining this "removal-driven" data scaling with a memory-efficient post-training algorithm called GRAR, the system achieves state-of-the-art performance in both removing and inserting objects while realistically handling complex interaction effects like shadows, reflections, and lighting.

What Problem Does This Paper Address?

Video object editing—specifically removal (erasing an object) and injection (inserting a new one)—is significantly harder than image editing. A realistic edit must account for the "dynamic imprint" an object leaves on its environment: the way a person's feet deform a trampoline, the shadow a camel casts on a dune, or the reflection of a car on a wet floor.

The primary bottleneck is supervision. To train a model to "inject" an object, you ideally need perfectly aligned "before and after" video pairs of the same scene. Collecting such pairs in the real world is nearly impossible, and synthetic data often lacks the photorealism and complex physics of real footage. While removal can be bootstrapped from video inpainting (filling in holes), injection requires the model to hallucinate plausible motion and environmental interactions from scratch.

Methods

The authors propose a three-stage pipeline to bridge this gap:

  1. Effect-Aware Video Object Removal: They first train a robust "remover" model. It starts with a strong video inpainting prior and is then fine-tuned on limited real-world pairs to ensure it erases not just the object, but also its associated shadows and reflections.
  2. Removal-Driven Injection Pre-training: This is the "aha" moment. They apply their trained remover to 69,000 real-world videos to create "object-absent" versions. By reversing these pairs, they create a massive, high-quality dataset where the "ground truth" is a real video with natural physics and lighting.
  3. Post-Training with GRAR: Pixel-level reconstruction loss (Stage 2) often fails to capture "perceptual realism"—it might produce blurry motion or inconsistent colors. To fix this, they use Reinforcement Learning from Human Feedback (RLHF). However, standard RL methods like PPO require keeping a "frozen" reference model in memory, which is impossible for a 23B parameter model on standard hardware. They introduce Group-Relative Advantage Regression (GRAR), a VRAM-efficient algorithm that uses a single trainable policy and relative ranking to align the model with human preferences without the memory overhead of a reference model.
The UniVOE training workflow: Stage 1 builds the remover, Stage 2 uses it to scale data for the injector, and Stage 3 refines the injector via GRAR.

Datasets

  • Training: The authors curated a massive dataset including 63K videos for inpainting pre-training, 68K open-source object-present/absent pairs, and 69K removal-derived pairs generated by their own Stage 1 model.
  • Evaluation: They benchmarked against ROSE (56 pairs) and VOR (100 pairs), which are standard datasets for video object removal and side-effect erasing.

Key Insights and Ideas

  • Reciprocal Asymmetry: The core philosophy is that removal and injection are two sides of the same coin, but removal is easier to learn. By mastering removal first, you can "unlock" the data needed for injection.
  • Interaction Awareness: Unlike generic inpainters that only fill a masked area, UniVOE is specifically trained to look outside the mask to find and erase (or create) shadows and reflections.
  • The "Memory Wall" Solution: GRAR is a significant contribution for researchers working with large video models. By discarding the reference policy and using in-group median-splitting for advantages, they made online RL feasible for a 23B parameter model on a single 80GB VRAM GPU.
Figure 1: We propose UniVOE, a video object editing pipeline that enables object-centered removal and injection, together with predictions of changes in object-related effects. Such as footprint, shadow, reflection, and lighting. Users can easily remove or insert target objects using prompts, as shown above each editing result.

Key Results

UniVOE outperforms both top-tier open-source models (like VideoPainter and VACE) and commercial systems (like Kling3.0 and Seedance2.0).

  • Quantitative: On the VOR benchmark, UniVOE achieved a PSNR of 23.752 and an FVD (Fréchet Video Distance) of 146.37, significantly lower (better) than competitors, indicating higher temporal consistency and realism.
  • Qualitative: As seen in Figure 4, while other models often leave "ghost" shadows or produce unnatural textures, UniVOE successfully recovers physically realistic scenes, such as the complex lighting of a car in a garage or the flow of milk into a glass.
Figure 4: Visual comparison on video object removal. Other methods either remain object-related effects traces (e.g., lighting of the car), or predict unnatural background contents. In contrast, our method removes object itself and recovers physically realistic scene. It outperforms both open-source and commercial models. More visual comparison can be found in supplementary.

Limitations and Future Work

  • Bootstrap Dependence: The injector's quality is inherently capped by the remover's performance. If the remover leaves a tiny artifact in the "absent" video, the injector might learn to treat that artifact as a cue.
  • Reward Model Fidelity: The success of the GRAR stage depends on the quality of the underlying reward model (a fine-tuned Qwen3-VL). Improving video-specific reward models is a key direction for future research.
  • Single Backbone: Currently, the framework is tied to a specific 23B backbone (Wan2.1). Testing the "removal-driven" recipe across different architectures remains to be explored.

Summary updated

Abstract

Video object removal and injection are reciprocal editing operations between object-present and object-absent video states, yet they differ substantially in training difficulty. Removal can be effectively bootstrapped from video inpainting and further adapted to erase object-related interaction effects. Injection, however, requires large-scale paired supervision to model object appearance, dynamics, and scene-dependent effects, while perfectly aligned real-world pairs are difficult to collect and synthetic data lacks sufficient realism and diversity. To address this challenge, we present UniVOE, a removal-driven framework for training both video object removal and injection models. We first train an effect-aware remover by transferring video inpainting priors and finetuning on object-present/object-absent pairs. The trained remover converts real object-present videos into object-absent counterparts, creating large-scale paired supervision for injection. An injection model is trained on these pairs in the reverse direction to recover objects together with their dynamics and interaction effects. Moreover, since pixel-level reconstruction alone cannot adequately capture perceptual realism, we introduce a reward-based post-training method that aligns injection results with human preferences using a proprietary reward model. Experiments against strong open-source and commercial systems demonstrate that UniVOE achieves state-of-the-art performance on both object removal and injection, particularly in challenging interaction-effect cases.
BibTeX
@misc{yang2026univoe,
  author = {Shuzhou Yang and Lanqing Guo and Lingen Li and Weiqi Li and Zhao Zhong and Liefeng Bo and Jian Zhang},
  title = {{UniVOE: A Removal-Driven Framework for Interaction-Aware Universal Video Object Editing}},
  year = {2026},
  eprint = {2610.31641},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.31641},
}
Figure