This paper introduces a paradigm shift in video editing: language-driven video inpainting. Instead of painstakingly drawing pixel-perfect masks to remove objects from a video, users can simply tell the computer what to do using natural language. To enable this, the authors present a massive new dataset, ROVI, and a novel diffusion-based framework, LGVI, that leverages Multimodal Large Language Models (MLLMs) to turn complex conversational requests into seamless video edits.
TL;DR
The authors propose a new task—language-driven video inpainting—that replaces manual binary masks with natural language instructions. They introduce the ROVI dataset (5,650 videos) and the LGVI framework, the first end-to-end baseline that integrates MLLMs to handle both simple referring expressions and complex, interactive chat-style editing requests.
What Problem Does This Paper Address?
Traditional video inpainting is a "mask-centric" process. To remove an object, a user must provide a binary mask for every frame, which is incredibly labor-intensive and impractical for long videos. While automatic segmentation tools exist, they often fail or require manual cleanup.
Furthermore, existing language-driven image editing models often struggle with video, failing to maintain temporal consistency or misinterpreting instructions when multiple similar objects are present. There is a clear gap: we lack a unified, end-to-end system that can understand "remove the woman on the left" or "make this scene look less crowded" and execute that edit across a video sequence without manual masking.
Methods
The authors propose a two-tier framework:
- LGVI (Language-Driven Video Inpainting): This is the base model, built on a diffusion architecture (Stable Diffusion). To handle video, the authors "inflate" the 2D U-Net by adding a temporal dimension and a parameter-efficient Temporal Attention module. This ensures that the inpainted areas look consistent from frame to frame.
- LGVI-I (Interactive): This extension integrates a Multimodal Large Language Model (specifically LLaVA-7B) to act as a reasoning engine. For complex "Interactive Video Inpainting" tasks, the MLLM processes the video frames and the user's chat history to deduce the user's intent. It generates a text response and a set of special "prompt tokens" that guide the diffusion model's cross-attention layers.
A key technical detail is the use of a Mask Decoder during training. While the model doesn't need masks at inference time, predicting them during training provides an explicit supervision signal that helps the model learn exactly where to apply the inpainting based on the text.
Datasets
The authors introduce the Remove Objects from Videos by Instructions (ROVI) dataset. It consists of:
- 5,650 videos and 9,091 inpainting results.
- Triplets of data: the original video, the language instruction, and the ground-truth inpainted video.
- Two sub-tasks: "Referring" (simple commands) and "Interactive" (complex dialogues).
To build ROVI, they repurposed existing segmentation datasets (Refer-YouTube-VOS and A2D-Sentences) and used state-of-the-art inpainting models (E2FGVI) combined with human selection to generate high-quality ground truths. For the interactive part, they used LLMs to simulate realistic, multi-turn user conversations.
Key Insights and Ideas
- Reasoning vs. Execution: By splitting the task between an MLLM (reasoning) and a Diffusion Model (execution), the system can handle "implicit" requests. If a user says "make the highway less crowded," the MLLM identifies the "white sports car" as the object to remove, a feat simple referring models cannot achieve.
- The Power of Mask Supervision: One of the most important findings was that training the model to predict a mask—even if that mask is never used by the end-user—is critical for performance. Without this "mask supervision," the model often fails to localize the correct object.
- Temporal Inflation is Efficient: Adding temporal attention modules between existing spatial layers allows the model to learn video dynamics without needing to retrain the entire massive U-Net from scratch.
Key Results
LGVI represents the first end-to-end baseline for this task. Quantitatively, it outperforms general image editing models like InstructPix2Pix and MagicBrush across all video metrics (PSNR, SSIM, VFID).
- Referring Task: LGVI achieved a PSNR of 22.85, significantly higher than MagicBrush's 20.39.
- Interactive Task: The MLLM-enhanced LGVI-I reached a PSNR of 22.24, proving its ability to handle complex reasoning better than two-stage baselines.
Qualitatively, the model shows impressive robustness. As seen in Figure 8, while other models might remove the wrong object or fail to understand a conversational request, LGVI-I correctly identifies and removes the intended target.
Limitations and Future Work
Despite its strengths, the model has clear areas for improvement:
- Ambiguity: Very vague descriptions (e.g., "the second duck from the left" in a crowded pond) still lead to failures.
- Spatial Precision: The current diffusion backbone sometimes lacks a "fine-grained" understanding of exact spatial coordinates, leading to slightly misplaced edits in complex scenes.
- Inference Speed: Like all diffusion models, LGVI is relatively slow due to the iterative denoising process, making real-time interactive editing a challenge for future research.
Summary updated