The development of Unified Multimodal Models (UMMs)—systems that can both "see" (understand) and "draw" (generate)—is currently hitting a data wall. While we have massive datasets for image captioning or text-to-image generation, we lack high-quality, structured data that teaches a model how to reason through a sequence of visual changes.
A new paper from researchers at Zhejiang University and the University of Hong Kong introduces VidUEU-Agent, an autonomous pipeline designed to mine raw video for the "missing link" of multimodal instruction data.
TL;DR
VidUEU-Agent is an automated framework that transforms raw, unstructured video into high-quality instruction data for multimodal understanding, editing, and unified tasks. By using an agentic feedback loop to refine its own data distribution, the system produced VidUEU-100K, a dataset that significantly boosts the performance of state-of-the-art models across diverse benchmarks.
What Problem Does This Paper Address?
The "Multimodal" field has traditionally moved along two separate tracks: Understanding (e.g., LLaVA) and Generation (e.g., Stable Diffusion). Unified models try to do both, but they require a specific type of data: interleaved sequences where the model must understand a visual state, process an instruction, and then predict or generate the next visual state.
Manually annotating these sequences is prohibitively expensive. Raw video is an obvious alternative source, as it inherently contains temporal dynamics and causal interactions. However, video is "noisy"—full of scene cuts, blurry frames, and redundant information. The challenge is building a system that can autonomously filter this noise and synthesize precise, multi-step instructions.
Methods
The authors propose a four-stage agentic pipeline that treats data curation as a planning and tool-use problem:
- Global Perception: The agent splits videos into shots and generates summaries at the shot, chapter, and video levels. It uses a "Gate A" filter to immediately discard low-quality or uninformative shots.
- Keyframe Mining: For the remaining shots, the agent plans which specialized tools (like OCR, ASR, or object detectors) are needed to extract the most relevant signals. It then uses a facility-location greedy selection algorithm to pick the most representative keyframes.
- Metadata Construction: The agent cleans the extracted signals and models the "transitions"—describing exactly what changed between keyframes (e.g., "the cat shifts from a calm pose to a wide-open mouth").
- Instruction Data Construction: Finally, the agent routes this metadata into specific task templates (Understanding, Editing, or Unified) and synthesizes the final training samples.
Crucially, the system includes an Autonomous Data Iteration loop. It trains a model on a subset of data, evaluates it on benchmarks, and then "reflects" on the results to adjust its sampling strategy (e.g., "we need more sports data" or "the editing tasks are too easy") for the next round of curation.
Datasets
The researchers scaled this pipeline to create VidUEU-100K, derived from over 8,000 raw videos. The dataset includes:
- 40K Understanding samples: Fine-grained perception and visual reasoning.
- 30K Editing samples: Instruction-based image modification.
- 30K Unified samples: Interleaved text-image tasks requiring both understanding and generation.
The pipeline is remarkably effective at filtering out "junk" data that would otherwise degrade model performance, as shown in their failure case analysis:
Key Insights and Ideas
- Video as a State-Transition Teacher: The core "aha" moment is that video isn't just a series of images; it's a record of state changes. By modeling these transitions, the agent creates data that teaches models the "logic" of visual editing and sequential reasoning.
- The Power of "Gates": The multi-stage filtering (Gates A, B, and C) is vital. Ablation studies showed that removing even one gate—especially the frame quality filter—led to a significant drop in the final model's performance.
- Traceability via Citations: To reduce hallucinations, the agent is required to "cite" the specific tool signals (like OCR text or bounding boxes) it used to generate an instruction. This ensures the synthetic data is grounded in verifiable evidence.
Key Results
The authors fine-tuned several baseline models, including InternVL3-8B and Bagel-7B-MoT, on VidUEU-100K. The results were consistently positive across the board:
- Understanding: InternVL3-8B saw its score on the MMDU benchmark (which tests multi-image reasoning) jump from 53.1 to 61.1.
- Editing: On GEdit-Bench, their fine-tuned Bagel model outperformed specialized editing models like Instruct-Pix2Pix and MagicBrush.
- Unified Tasks: On the WEAVEbench benchmark, which evaluates interleaved comprehension and generation, the fine-tuned model improved from an overall score of 0.446 to 0.501.
Limitations and Future Work
While VidUEU-Agent is a major step forward, the authors note a few hurdles:
- Pure Generation: The pipeline is optimized for transitions and editing, making it less suitable for curating "pure" text-to-image data (where there is no source image).
- Long-Horizon Reasoning: While the model excels at short sequences, long-horizon interleaved interactions (many turns of images and text) remain an open challenge.
- Scale: 100K samples is a solid start, but the authors aim to scale to millions of samples to further improve data diversity and long-tail coverage.
For researchers looking to bridge the gap between seeing and doing, VidUEU-Agent provides a reproducible, scalable blueprint for turning the chaos of the web's video content into a structured curriculum for the next generation of AI.
Summary updated