VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks

Tengju Ru, Weitong Lian, Zecong Tang, Lingyi Meng, Haoran Li, Zhejun Cui, Yichen Zhu, Hangshuo Cao, Qi Kang, Yechi Liu, Kaixuan Wang, Yu-Jie Yuan, Chunwei Wang, Yu Zhang, Bo DaiarXiv 2026 · arXiv

The development of Unified Multimodal Models (UMMs)—systems that can both "see" (understand) and "draw" (generate)—is currently hitting a data wall. While we have massive datasets for image captioning or text-to-image generation, we lack high-quality, structured data that teaches a model how to reason through a sequence of visual changes.

A new paper from researchers at Zhejiang University and the University of Hong Kong introduces VidUEU-Agent, an autonomous pipeline designed to mine raw video for the "missing link" of multimodal instruction data.

TL;DR

VidUEU-Agent is an automated framework that transforms raw, unstructured video into high-quality instruction data for multimodal understanding, editing, and unified tasks. By using an agentic feedback loop to refine its own data distribution, the system produced VidUEU-100K, a dataset that significantly boosts the performance of state-of-the-art models across diverse benchmarks.

What Problem Does This Paper Address?

The "Multimodal" field has traditionally moved along two separate tracks: Understanding (e.g., LLaVA) and Generation (e.g., Stable Diffusion). Unified models try to do both, but they require a specific type of data: interleaved sequences where the model must understand a visual state, process an instruction, and then predict or generate the next visual state.

Manually annotating these sequences is prohibitively expensive. Raw video is an obvious alternative source, as it inherently contains temporal dynamics and causal interactions. However, video is "noisy"—full of scene cuts, blurry frames, and redundant information. The challenge is building a system that can autonomously filter this noise and synthesize precise, multi-step instructions.

Methods

The authors propose a four-stage agentic pipeline that treats data curation as a planning and tool-use problem:

  1. Global Perception: The agent splits videos into shots and generates summaries at the shot, chapter, and video levels. It uses a "Gate A" filter to immediately discard low-quality or uninformative shots.
  2. Keyframe Mining: For the remaining shots, the agent plans which specialized tools (like OCR, ASR, or object detectors) are needed to extract the most relevant signals. It then uses a facility-location greedy selection algorithm to pick the most representative keyframes.
  3. Metadata Construction: The agent cleans the extracted signals and models the "transitions"—describing exactly what changed between keyframes (e.g., "the cat shifts from a calm pose to a wide-open mouth").
  4. Instruction Data Construction: Finally, the agent routes this metadata into specific task templates (Understanding, Editing, or Unified) and synthesizes the final training samples.
Figure 2: Overview of VidUEU-Agent. The agent performs global perception with quality gating, then conducts keyframe mining using multimodal signals and tool verification, builds metadata, and finally synthesizes instruction data for understanding, editing, and unified tasks.

Crucially, the system includes an Autonomous Data Iteration loop. It trains a model on a subset of data, evaluates it on benchmarks, and then "reflects" on the results to adjust its sampling strategy (e.g., "we need more sports data" or "the editing tasks are too easy") for the next round of curation.

Datasets

The researchers scaled this pipeline to create VidUEU-100K, derived from over 8,000 raw videos. The dataset includes:

  • 40K Understanding samples: Fine-grained perception and visual reasoning.
  • 30K Editing samples: Instruction-based image modification.
  • 30K Unified samples: Interleaved text-image tasks requiring both understanding and generation.

The pipeline is remarkably effective at filtering out "junk" data that would otherwise degrade model performance, as shown in their failure case analysis:

Figure 3: Failure Case Examples: Illustrating four common rejected cases in the processing pipeline: Global Rejection, Quality Failure, Gen-Unsuitability, and Task Mismatch.

Key Insights and Ideas

  • Video as a State-Transition Teacher: The core "aha" moment is that video isn't just a series of images; it's a record of state changes. By modeling these transitions, the agent creates data that teaches models the "logic" of visual editing and sequential reasoning.
  • The Power of "Gates": The multi-stage filtering (Gates A, B, and C) is vital. Ablation studies showed that removing even one gate—especially the frame quality filter—led to a significant drop in the final model's performance.
  • Traceability via Citations: To reduce hallucinations, the agent is required to "cite" the specific tool signals (like OCR text or bounding boxes) it used to generate an instruction. This ensures the synthetic data is grounded in verifiable evidence.

Key Results

The authors fine-tuned several baseline models, including InternVL3-8B and Bagel-7B-MoT, on VidUEU-100K. The results were consistently positive across the board:

  • Understanding: InternVL3-8B saw its score on the MMDU benchmark (which tests multi-image reasoning) jump from 53.1 to 61.1.
  • Editing: On GEdit-Bench, their fine-tuned Bagel model outperformed specialized editing models like Instruct-Pix2Pix and MagicBrush.
  • Unified Tasks: On the WEAVEbench benchmark, which evaluates interleaved comprehension and generation, the fine-tuned model improved from an overall score of 0.446 to 0.501.
Table 1: Comparison of fine-tuned InternVL3-8B & Bagel, and baseline models on benchmarks.

Limitations and Future Work

While VidUEU-Agent is a major step forward, the authors note a few hurdles:

  • Pure Generation: The pipeline is optimized for transitions and editing, making it less suitable for curating "pure" text-to-image data (where there is no source image).
  • Long-Horizon Reasoning: While the model excels at short sequences, long-horizon interleaved interactions (many turns of images and text) remain an open challenge.
  • Scale: 100K samples is a solid start, but the authors aim to scale to millions of samples to further improve data diversity and long-tail coverage.

For researchers looking to bridge the gap between seeing and doing, VidUEU-Agent provides a reproducible, scalable blueprint for turning the chaos of the web's video content into a structured curriculum for the next generation of AI.

Summary updated

Abstract

The advancement of multimodal learning is heavily constrained by the scarcity and high annotation cost of high-quality instruction data. While raw videos offer an abundant and dynamic source of visual knowledge, transforming them into precise instruction data remains highly challenging. To bridge this gap, we present VidUEU-Agent, an automated pipeline that synthesizes instruction data from raw videos for multimodal understanding, editing, and unified tasks. VidUEU-Agent follows a four-stage pipeline: it performs global perception to filter low-quality shots, mines keyframes and multimodal signals, constructs metadata and routes the metadata to synthesize instruction samples. More importantly, our agent framework can autonomously iterate the data distribution based on downstream training results, thereby optimizing data quality, balance, and training effectiveness. Scaling this pipeline, we construct VidUEU-100K, a large-scale dataset of 100K training samples spanning multiple domains and tasks, providing a robust foundation for developing comprehensive multimodal capabilities. Extensive experiments demonstrate that fine-tuning baseline models on VidUEU-100K yields significant and consistent performance gains across diverse representative benchmarks, validating the effectiveness of our proposed agent framework and dataset.
BibTeX
@misc{ru2026vidueu,
  author = {Tengju Ru and Weitong Lian and Zecong Tang and Lingyi Meng and Haoran Li and Zhejun Cui and Yichen Zhu and Hangshuo Cao and Qi Kang and Yechi Liu and Kaixuan Wang and Yu-Jie Yuan and Chunwei Wang and Yu Zhang and Bo Dai},
  title = {{VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks}},
  year = {2026},
  eprint = {2611.04418},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04418},
}
Figure