VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation

Shoubin Yu, Difan Liu, Ziqiao Ma, Yicong Hong, Yang Zhou, Hao Tan, Joyce Chai, Mohit Bansal

TL;DR

VEGGIE is a unified, end-to-end framework that seamlessly blends video concept editing, grounding, and reasoning under a single diffusion loss. By combining a Multimodal Large Language Model (MLLM) with a Video Diffusion Model (VidDM) via continuous task tokens, it handles diverse user instructions—ranging from object addition and removal to complex reasoning segmentation—without relying on intermediate masks or layouts.

What Problem Does This Paper Address?

While text-to-video diffusion models have made massive leaps, instructional video editing (e.g., "add a hat to the girl" or "replace the dog with a cat") remains highly fragmented. Current state-of-the-art approaches suffer from three core limitations:

  1. Lack of End-to-End Unity: Most models rely on clunky multi-step pipelines requiring intermediate layout generation, user-provided masks, or captioning models.
  2. Complex Optimization: Connecting MLLMs to diffusion models typically requires balancing multiple training objectives, such as separate language losses and mask losses, leading to hyperparameter headaches.
  3. Poor Multitasking: Existing models are usually specialized; a model that excels at global stylization often fails at local object removal, and almost none can handle complex reasoning queries (e.g., "What can be used to contain food? Highlight it.").

VEGGIE addresses these gaps by formulating video editing, grounding, and reasoning as a single, unified video-to-video task optimized purely via pixel-space diffusion loss.

Figure 1. We propose VEGGIE, a unified and versatile video generative model that handles various tasks for both video concept grounding and editing according to user instructions. With VEGGIE, users can locate, add, delete, and change concepts in a given video through diverse instruction formats (direct referring instruction or reasoning-demanding questions). Users can also edit videos with multimodal instruction empowered by MLLM, enabling applications like video editing from a reference image.

Methods

VEGGIE's architecture consists of four main components:

  • An MLLM (LLaVA-OneVision-7B): Interprets the user's text instruction alongside the input video frames.
  • Continuous Grounded Task Queries: Instead of outputting discrete text tokens (which block gradient propagation), the MLLM generates continuous, learnable task query embeddings for each frame.
  • An Alignment Network: A single-layer MLP that projects the MLLM's query embeddings directly into the conditioning space of the diffusion model.
  • A Video Diffusion Model (VidDM): Initialized from an instructional image-editing model (MagicBrush) and inflated with temporal attention layers to ensure smooth video generation.
The end-to-end architecture of VEGGIE, mapping input videos and instructions to pixel-space outputs.

To train this complex system without model collapse, the authors introduce a two-stage curriculum learning strategy:

  1. Stage 1 (Image Pre-alignment): The MLLM is frozen, and the alignment network and diffusion model are trained on massive image-editing datasets to align the language and pixel spaces.
  2. Stage 2 (Video Fine-tuning): The entire pipeline (including the MLLM via LoRA) is fine-tuned end-to-end on multi-task video data to master temporal consistency and dynamics.

Additionally, the authors developed an automated Image-to-Video (I2V) data synthesis pipeline to turn high-quality static image-editing pairs into temporally consistent video-editing training data.

Datasets

VEGGIE was trained on a massive blend of data:

  • Image Data (~3.4M pairs): Sourced from datasets like Seed-Data-Edit, LISA, gRefCoCo, and MagicBrush for initial alignment.
  • Video Data (~136K pairs): Included ROVI, VPLM, InstructV2V, and the authors' newly synthesized VEG-Edit dataset.

For evaluation, the authors introduced VEG-Bench, a comprehensive benchmark spanning 8 distinct editing skills (concept addition, removal, object changing, background changing, visual feature changing, stylization, object grounding, and reasoning segmentation) consisting of 132 meticulously annotated video-instruction pairs.

Key Insights and Ideas

  • Multi-Task Synergy: The authors discovered that learning to locate an object (grounding) and learning to modify it (editing) are mutually reinforcing. Training on both tasks simultaneously yields lower Fréchet Video Distance (FVD) and higher structural similarity (SSIM) scores, provided the data mixture is well-balanced.
  • The Power of Continuous Tokens: By avoiding discrete text tokens as an intermediate representation between the MLLM and the diffusion model, VEGGIE maintains a fully differentiable pipeline, simplifying optimization down to a single diffusion loss.
  • Emergent Properties: Without explicit training, VEGGIE demonstrates zero-shot multimodal instruction following (e.g., performing style transfer or object addition using a reference image) and few-shot in-context learning from image demonstration pairs.
Emergent zero-shot multimodal instruction following in VEGGIE using reference images.
Figure 3. In-context editing emerges in VEGGIE, allowing for few-shot learning of editing tasks with paired image demonstrations.

Key Results

  • VEG-Bench Superiority: VEGGIE achieved the best average ranking (1.78) across all 8 editing skills, outperforming specialized instructional baselines like InsV2V (2.00) and InstructDiff (3.00).
  • Grounding & Reasoning Excellence: On video object grounding and reasoning segmentation tasks, VEGGIE outperformed standard generative editing models by a wide margin (e.g., achieving a Jaccard Index of 37.74 in grounding compared to InsV2V's 13.89).
  • Qualitative Precision: Visual results demonstrate that VEGGIE successfully executes localized edits (like removing a specific person) without altering surrounding contexts or introducing severe temporal artifacts.

Limitations and Future Work

Because VEGGIE is built on top of the older Stable Diffusion 1.5 backbone, its raw visual generation quality lags behind newer Diffusion Transformer (DiT) or flow-based architectures. The generated videos are also relatively short. Furthermore, the authors noted that an excessive amount of grounding data can introduce unwanted visual artifacts, meaning that fine-tuning the exact data mixture strategy remains an open challenge. Future work will focus on scaling VEGGIE to modern DiT backbones and extending video generation lengths.

Summary updated

Abstract

Recent video diffusion models have enhanced video editing, but it remains challenging to handle instructional editing and diverse tasks (e.g., adding, removing, changing) within a unified framework. In this paper, we introduce VEGGIE, a Video Editor with Grounded Generation from Instructions, a simple end-to-end framework that unifies video concept editing, grounding, and reasoning based on diverse user instructions. Specifically, given a video and text query, VEGGIE first utilizes an MLLM to interpret user intentions in instructions and ground them to the video contexts, generating frame-specific grounded task queries for pixel-space responses. A diffusion model then renders these plans and generates edited videos that align with user intent. To support diverse tasks and complex instructions, we employ a curriculum learning strategy: first aligning the MLLM and video diffusion model with large-scale instructional image editing data, followed by end-to-end fine-tuning on high-quality multitask video data. Additionally, we introduce a novel data synthesis pipeline to generate paired instructional video editing data for model training. It transforms static image data into diverse, high-quality video editing samples by leveraging Image-to-Video models to inject dynamics. VEGGIE shows strong performance in instructional video editing with different editing skills, outperforming the best instructional baseline as a versatile model, while other models struggle with multi-tasking. VEGGIE also excels in video object grounding and reasoning segmentation, where other baselines fail. We further reveal how the multiple tasks help each other and highlight promising applications like zero-shot multimodal instructional and in-context video editing.
BibTeX
@inproceedings{yu2025veggie,
  author = {Shoubin Yu and Difan Liu and Ziqiao Ma and Yicong Hong and Yang Zhou and Hao Tan and Joyce Chai and Mohit Bansal},
  title = {{VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=iD8U1ZJC8M},
}
Figure