TL;DR
UniVOE is a novel framework that solves the "data scarcity" problem in video object editing by using a high-quality object remover to generate a massive synthetic-but-realistic training dataset for an object injector. By combining this "removal-driven" data scaling with a memory-efficient post-training algorithm called GRAR, the system achieves state-of-the-art performance in both removing and inserting objects while realistically handling complex interaction effects like shadows, reflections, and lighting.
What Problem Does This Paper Address?
Video object editing—specifically removal (erasing an object) and injection (inserting a new one)—is significantly harder than image editing. A realistic edit must account for the "dynamic imprint" an object leaves on its environment: the way a person's feet deform a trampoline, the shadow a camel casts on a dune, or the reflection of a car on a wet floor.
The primary bottleneck is supervision. To train a model to "inject" an object, you ideally need perfectly aligned "before and after" video pairs of the same scene. Collecting such pairs in the real world is nearly impossible, and synthetic data often lacks the photorealism and complex physics of real footage. While removal can be bootstrapped from video inpainting (filling in holes), injection requires the model to hallucinate plausible motion and environmental interactions from scratch.
Methods
The authors propose a three-stage pipeline to bridge this gap:
- Effect-Aware Video Object Removal: They first train a robust "remover" model. It starts with a strong video inpainting prior and is then fine-tuned on limited real-world pairs to ensure it erases not just the object, but also its associated shadows and reflections.
- Removal-Driven Injection Pre-training: This is the "aha" moment. They apply their trained remover to 69,000 real-world videos to create "object-absent" versions. By reversing these pairs, they create a massive, high-quality dataset where the "ground truth" is a real video with natural physics and lighting.
- Post-Training with GRAR: Pixel-level reconstruction loss (Stage 2) often fails to capture "perceptual realism"—it might produce blurry motion or inconsistent colors. To fix this, they use Reinforcement Learning from Human Feedback (RLHF). However, standard RL methods like PPO require keeping a "frozen" reference model in memory, which is impossible for a 23B parameter model on standard hardware. They introduce Group-Relative Advantage Regression (GRAR), a VRAM-efficient algorithm that uses a single trainable policy and relative ranking to align the model with human preferences without the memory overhead of a reference model.
Datasets
- Training: The authors curated a massive dataset including 63K videos for inpainting pre-training, 68K open-source object-present/absent pairs, and 69K removal-derived pairs generated by their own Stage 1 model.
- Evaluation: They benchmarked against ROSE (56 pairs) and VOR (100 pairs), which are standard datasets for video object removal and side-effect erasing.
Key Insights and Ideas
- Reciprocal Asymmetry: The core philosophy is that removal and injection are two sides of the same coin, but removal is easier to learn. By mastering removal first, you can "unlock" the data needed for injection.
- Interaction Awareness: Unlike generic inpainters that only fill a masked area, UniVOE is specifically trained to look outside the mask to find and erase (or create) shadows and reflections.
- The "Memory Wall" Solution: GRAR is a significant contribution for researchers working with large video models. By discarding the reference policy and using in-group median-splitting for advantages, they made online RL feasible for a 23B parameter model on a single 80GB VRAM GPU.
Key Results
UniVOE outperforms both top-tier open-source models (like VideoPainter and VACE) and commercial systems (like Kling3.0 and Seedance2.0).
- Quantitative: On the VOR benchmark, UniVOE achieved a PSNR of 23.752 and an FVD (Fréchet Video Distance) of 146.37, significantly lower (better) than competitors, indicating higher temporal consistency and realism.
- Qualitative: As seen in Figure 4, while other models often leave "ghost" shadows or produce unnatural textures, UniVOE successfully recovers physically realistic scenes, such as the complex lighting of a car in a garage or the flow of milk into a glass.
Limitations and Future Work
- Bootstrap Dependence: The injector's quality is inherently capped by the remover's performance. If the remover leaves a tiny artifact in the "absent" video, the injector might learn to treat that artifact as a cue.
- Reward Model Fidelity: The success of the GRAR stage depends on the quality of the underlying reward model (a fine-tuned Qwen3-VL). Improving video-specific reward models is a key direction for future research.
- Single Backbone: Currently, the framework is tied to a specific 23B backbone (Wan2.1). Testing the "removal-driven" recipe across different architectures remains to be explored.
Summary updated