Abstract
Recent video diffusion models have enhanced video editing, but it remains challenging to handle instructional editing and diverse tasks (e.g., adding, removing, changing) within a unified framework. In this paper, we introduce VEGGIE, a Video Editor with Grounded Generation from Instructions, a simple end-to-end framework that unifies video concept editing, grounding, and reasoning based on diverse user instructions. Specifically, given a video and text query, VEGGIE first utilizes an MLLM to interpret user intentions in instructions and ground them to the video contexts, generating frame-specific grounded task queries for pixel-space responses. A diffusion model then renders these plans and generates edited videos that align with user intent. To support diverse tasks and complex instructions, we employ a curriculum learning strategy: first aligning the MLLM and video diffusion model with large-scale instructional image editing data, followed by end-to-end fine-tuning on high-quality multitask video data. Additionally, we introduce a novel data synthesis pipeline to generate paired instructional video editing data for model training. It transforms static image data into diverse, high-quality video editing samples by leveraging Image-to-Video models to inject dynamics. VEGGIE shows strong performance in instructional video editing with different editing skills, outperforming the best instructional baseline as a versatile model, while other models struggle with multi-tasking. VEGGIE also excels in video object grounding and reasoning segmentation, where other baselines fail. We further reveal how the multiple tasks help each other and highlight promising applications like zero-shot multimodal instructional and in-context video editing.
Introduction
While image editing has seen transformative progress, video editing remains a fragmented field. Most existing tools are specialized for specific tasks—such as stylization or object removal—and often require manual intermediate inputs like masks, layouts, or exhaustive captions. This fragmentation creates a barrier for users who wish to perform complex, instructional edits through natural language.
The fundamental challenge in instructional video editing lies in the intersection of multimodal reasoning and precise spatial-temporal grounding. A model must not only understand what a user wants to change (often from vague or complex prompts) but also exactly where and when that change should occur across the video sequence. VEGGIE (Video Editor with Grounded Generation from Instructions) addresses these challenges by providing a unified, end-to-end framework capable of handling diverse tasks—from concept addition and removal to reasoning-based segmentation—within a single model.
The VEGGIE Framework
VEGGIE's architecture is designed to bridge the gap between high-level linguistic intent and low-level pixel manipulation. It consists of four primary components:
- Multimodal Large Language Model (MLLM): Utilizing LLaVA-OV as its core, the MLLM acts as the reasoning engine. It interprets the user's natural language instruction in the context of the input video to formulate a frame-by-frame execution plan.
- Grounded Task Query Tokens: Instead of relying on discrete text tokens, VEGGIE employs continuous, learnable embeddings. These tokens represent the "grounded task queries" that guide the generation process, allowing for better gradient flow and more nuanced spatial information transfer between the language and vision modules.
- Alignment Network: A simple Multi-Layer Perceptron (MLP) projects the MLLM's output into the condition space required by the diffusion model.
- Video Diffusion Model (VidDM): Based on a modified Stable Diffusion architecture, the VidDM renders the final edited video frames, ensuring temporal consistency and high visual fidelity.
Training and Data Synthesis
To overcome the scarcity of paired "before and after" video editing data, the researchers developed a novel data synthesis pipeline. This pipeline takes high-quality image editing pairs and "lifts" them into the video domain using Image-to-Video (I2V) models. This process injects realistic motion while maintaining the consistency of the edit across frames.
The model is trained using a curriculum learning strategy. First, the MLLM and VidDM are aligned using massive image-editing datasets (3.4M pairs). Subsequently, the model is fine-tuned on high-quality multitask video data (133.9K pairs), enabling it to master complex temporal dynamics and diverse editing skills.
Evaluation and Key Results
The performance of VEGGIE was evaluated using VEG-Bench, a comprehensive new benchmark covering eight distinct skills: concept addition, removal, object changing, background changing, visual feature changing, stylization, object grounding, and reasoning segmentation.
VEGGIE significantly outperformed six major baselines, achieving the best average ranking (1.78) across all skills. A standout result was in Reasoning Segmentation, where VEGGIE achieved a Jaccard Index of 22.53, nearly 33% higher than the next best baseline. This task requires the model to identify objects based on complex logic (e.g., "what can be used to contain food?") and generate precise masks, a capability where traditional generative models typically fail.
Beyond its primary tasks, VEGGIE demonstrated emergent abilities in zero-shot multimodal editing and in-context learning. As shown above, the model can effectively transfer a style from a single reference image to a target video without specific training for that style.
Conclusion
VEGGIE represents a significant step toward a truly versatile and instructional video editor. By unifying reasoning, grounding, and generation, it simplifies the user experience while expanding the scope of possible edits. While currently limited by the base Stable Diffusion 1.5 architecture and short clip lengths, future iterations utilizing Diffusion Transformers (DiT) and larger datasets promise even greater visual quality and temporal stability. The synergy discovered between grounding and editing tasks suggests that multi-tasking is a key path forward for more capable video foundation models.
Summary updated