Towards Language-Driven Video Inpainting via Multimodal Large Language Models

Jianzong Wu, Xiangtai Li, Chenyang Si, Shangchen Zhou, Jingkang Yang, Jiangning Zhang, Yining Li, Kai Chen, Yunhai Tong, Ziwei Liu, Chen Change Loy

This paper introduces a paradigm shift in video editing: language-driven video inpainting. Instead of painstakingly drawing pixel-perfect masks to remove objects from a video, users can simply tell the computer what to do using natural language. To enable this, the authors present a massive new dataset, ROVI, and a novel diffusion-based framework, LGVI, that leverages Multimodal Large Language Models (MLLMs) to turn complex conversational requests into seamless video edits.

TL;DR

The authors propose a new task—language-driven video inpainting—that replaces manual binary masks with natural language instructions. They introduce the ROVI dataset (5,650 videos) and the LGVI framework, the first end-to-end baseline that integrates MLLMs to handle both simple referring expressions and complex, interactive chat-style editing requests.

What Problem Does This Paper Address?

Traditional video inpainting is a "mask-centric" process. To remove an object, a user must provide a binary mask for every frame, which is incredibly labor-intensive and impractical for long videos. While automatic segmentation tools exist, they often fail or require manual cleanup.

Furthermore, existing language-driven image editing models often struggle with video, failing to maintain temporal consistency or misinterpreting instructions when multiple similar objects are present. There is a clear gap: we lack a unified, end-to-end system that can understand "remove the woman on the left" or "make this scene look less crowded" and execute that edit across a video sequence without manual masking.

Methods

The authors propose a two-tier framework:

  1. LGVI (Language-Driven Video Inpainting): This is the base model, built on a diffusion architecture (Stable Diffusion). To handle video, the authors "inflate" the 2D U-Net by adding a temporal dimension and a parameter-efficient Temporal Attention module. This ensures that the inpainted areas look consistent from frame to frame.
  2. LGVI-I (Interactive): This extension integrates a Multimodal Large Language Model (specifically LLaVA-7B) to act as a reasoning engine. For complex "Interactive Video Inpainting" tasks, the MLLM processes the video frames and the user's chat history to deduce the user's intent. It generates a text response and a set of special "prompt tokens" that guide the diffusion model's cross-attention layers.
The LGVI and LGVI-I architecture, showing how MLLMs are integrated into the diffusion process.

A key technical detail is the use of a Mask Decoder during training. While the model doesn't need masks at inference time, predicting them during training provides an explicit supervision signal that helps the model learn exactly where to apply the inpainting based on the text.

Datasets

The authors introduce the Remove Objects from Videos by Instructions (ROVI) dataset. It consists of:

  • 5,650 videos and 9,091 inpainting results.
  • Triplets of data: the original video, the language instruction, and the ground-truth inpainted video.
  • Two sub-tasks: "Referring" (simple commands) and "Interactive" (complex dialogues).

To build ROVI, they repurposed existing segmentation datasets (Refer-YouTube-VOS and A2D-Sentences) and used state-of-the-art inpainting models (E2FGVI) combined with human selection to generate high-quality ground truths. For the interactive part, they used LLMs to simulate realistic, multi-turn user conversations.

The ROVI dataset annotation pipeline, involving both automated generation and human selection.

Key Insights and Ideas

  • Reasoning vs. Execution: By splitting the task between an MLLM (reasoning) and a Diffusion Model (execution), the system can handle "implicit" requests. If a user says "make the highway less crowded," the MLLM identifies the "white sports car" as the object to remove, a feat simple referring models cannot achieve.
  • The Power of Mask Supervision: One of the most important findings was that training the model to predict a mask—even if that mask is never used by the end-user—is critical for performance. Without this "mask supervision," the model often fails to localize the correct object.
  • Temporal Inflation is Efficient: Adding temporal attention modules between existing spatial layers allows the model to learn video dynamics without needing to retrain the entire massive U-Net from scratch.

Key Results

LGVI represents the first end-to-end baseline for this task. Quantitatively, it outperforms general image editing models like InstructPix2Pix and MagicBrush across all video metrics (PSNR, SSIM, VFID).

  • Referring Task: LGVI achieved a PSNR of 22.85, significantly higher than MagicBrush's 20.39.
  • Interactive Task: The MLLM-enhanced LGVI-I reached a PSNR of 22.24, proving its ability to handle complex reasoning better than two-stage baselines.

Qualitatively, the model shows impressive robustness. As seen in Figure 8, while other models might remove the wrong object or fail to understand a conversational request, LGVI-I correctly identifies and removes the intended target.

Qualitative comparison showing LGVI-I successfully handling complex interactive requests where baselines fail.

Limitations and Future Work

Despite its strengths, the model has clear areas for improvement:

  • Ambiguity: Very vague descriptions (e.g., "the second duck from the left" in a crowded pond) still lead to failures.
  • Spatial Precision: The current diffusion backbone sometimes lacks a "fine-grained" understanding of exact spatial coordinates, leading to slightly misplaced edits in complex scenes.
  • Inference Speed: Like all diffusion models, LGVI is relatively slow due to the iterative denoising process, making real-time interactive editing a challenge for future research.

Summary updated

Abstract

We introduce a new task -- language-driven video inpainting, which uses natural language instructions to guide the inpainting process. This approach overcomes the limitations of traditional video inpainting methods that depend on manually labeled binary masks, a process often tedious and labor-intensive. We present the Remove Objects from Videos by Instructions (ROVI) dataset, containing 5,650 videos and 9,091 inpainting results, to support training and evaluation for this task. We also propose a novel diffusion-based language-driven video inpainting framework, the first end-to-end baseline for this task, integrating Multimodal Large Language Models to understand and execute complex language-based inpainting requests effectively. Our comprehensive results showcase the dataset's versatility and the model's effectiveness in various language-instructed inpainting scenarios. We will make datasets, code, and models publicly available.
BibTeX
@inproceedings{wu2024towards,
  author = {Jianzong Wu and Xiangtai Li and Chenyang Si and Shangchen Zhou and Jingkang Yang and Jiangning Zhang and Yining Li and Kai Chen and Yunhai Tong and Ziwei Liu and Chen Change Loy},
  title = {{Towards Language-Driven Video Inpainting via Multimodal Large Language Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure