UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalist

Zhengyang Liang, Daoan Zhang, Huichi Zhou, Rui Huang, Bobo Li, Yuechen Zhang, Shengqiong Wu, Xiaohan Wang, Jiebo Luo, Lizi Liao, Hao FeiarXiv 2025 · DOI · OpenReview

Introduction

UniVA is an open-source, multi-agent framework that unifies video understanding, generation, editing, and segmentation into a single, cohesive workflow. By employing a "Plan-and-Act" architecture and a hierarchical memory system, it enables complex, multi-step video creation—like generating a scene and then iteratively editing specific objects—that previously required manual coordination of multiple disparate AI tools.

The Challenge of Composite Video Workflows

While the AI field has seen an explosion of specialized models for video (e.g., models that only generate or only segment), real-world production requires composite workflows. For instance, a creator might want to generate a video from text, swap a character's outfit, and then extend the clip—all while maintaining visual consistency.

Current solutions fall into two camps, both with significant drawbacks:

  1. Single-task models: These are high-performing but isolated; they cannot reason across multiple steps without human "glue."
  2. Monolithic Video-Language Models (VLMs): These are often inflexible and difficult to extend with new specialized tools.

The authors identify a critical bottleneck: the lack of a unified system capable of long-horizon reasoning, proactive assistance, and seamless tool orchestration for video intelligence.

System Architecture: The Plan-and-Act Paradigm

UniVA introduces a dual-agent system designed for "Agentic Synergy." Its architecture is built on three pillars:

1. Plan-and-Act Architecture

  • Planner Agent: Interprets high-level user intent and decomposes it into a structured sequence of subtasks.
  • Executor Agent: Carries out these steps by invoking modular tools via the Model Context Protocol (MCP). This allows UniVA to be "omni-capable," easily plugging in new state-of-the-art models as tool servers.

2. Hierarchical Multi-level Memory

To sustain long-term context, UniVA uses three memory layers:

  • Global Memory: Stores persistent knowledge and historical traces to improve future planning.
  • Task Memory: Maintains intermediate artifacts (like segmentation masks or captions) so they can be reused across steps without recomputation.
  • User Memory: Tracks specific user preferences and styles for personalized interaction.

3. Self-Reflective Loops

The system doesn't just execute; it evaluates. If a generated video doesn't meet the requirements, the agent can initiate self-reflection to course-correct and adapt its plan. Because the system is agentic and modular, every step of the video production process is traceable, allowing for "any-conditioned" workflows (text-to-video, image-to-video, or video-to-video).

UniVA-Bench: Evaluating Agentic Intelligence

The authors introduce UniVA-Bench, a new benchmark suite designed to evaluate agentic video intelligence rather than just single-task accuracy. It leverages existing datasets like Video-MME (for understanding), Opens2v-nexus (for generation), and DAVIS2017 (for segmentation), but reformulates them into multi-step "goal cards."

Beyond standard quality scores (CLIP, DINO), it introduces Agentic Metrics:

  • Weighted Plan Edit Distance (wPED): Measures the accuracy of the generated plan against a ground truth.
  • Dependency Coverage (DepCov): Evaluates how well the agent handles tasks that depend on previous outputs.
  • Re-planning Quality (ReplanQ): Measures the agent's ability to recover from execution failures.

Key Insights and Results

A major insight from this work is the integration of perception and reasoning. UniVA uses its understanding module to help its perception tools. For example, if a "blue car" is occluded in a video, the agent can use narrative context to confirm it's the same car when it reappears, a task that pixel-level segmentation models often struggle with.

UniVA demonstrates superior performance across the board compared to both specialized models and general-purpose LLMs:

  • Understanding: On the LongVideo Understanding task, UniVA achieved an accuracy of 0.76, outperforming GPT-4o (0.52) and Gemini 2.5 Pro (0.65).
  • Segmentation: It significantly outperformed the SA2VA baseline, achieving a J&F score of 0.2467 vs. 0.1524.
  • Planning Robustness: The Plan-Act framework more than doubled the "Success Rate" (producing a structurally valid plan) compared to a single-agent setup (45.0% vs. 20.0%).
  • Human Preference: In a formal study, UniVA was the clear leader, winning in 4 out of 5 dimensions, including Semantic Accuracy and Style Consistency.

Conclusion

UniVA represents a significant step toward a universal video generalist. While it excels at complex instructions and narrative coherence, the authors acknowledge a current trade-off: it sometimes sacrifices strict frame-level similarity (DINO score) compared to specialized, non-agentic models. Future work will focus on optimizing the balance between high-level narrative control and fine-grained visual consistency, expanding the open-source tool ecosystem, and improving multi-agent communication efficiency.

Summary updated

Abstract

While specialized AI models excel at isolated video tasks like generation or understanding, real-world applications demand complex, iterative workflows that combine these capabilities. To bridge this gap, we introduce UniVA, an open-source, omni-capable multi-agent framework for next-generation video generalists that unifies video understanding, segmentation, editing, and generation into cohesive workflows. UniVA employs a Plan-and-Act dual-agent architecture that drives a highly automated and proactive workflow: a planner agent interprets user intentions and decomposes them into structured video-processing steps, while executor agents execute these through modular, MCP-based tool servers (for analysis, generation, editing, tracking, etc.). Through a hierarchical multi-level memory (global knowledge, task context, and user-specific preferences), UniVA sustains long-horizon reasoning, contextual continuity, and inter-agent communication, enabling interactive and self-reflective video creation with full traceability. This design enables iterative and any-conditioned video workflows (e.g., text/image/video-conditioned generation $\rightarrow$ multi-round editing $\rightarrow$ object segmentation $\rightarrow$ compositional synthesis) that were previously cumbersome to achieve with single-purpose models or monolithic video-language models. We also introduce UniVA-Bench, a benchmark suite of multi-step video tasks spanning understanding, editing, segmentation, and generation, to rigorously evaluate such agentic video systems. Both UniVA and UniVA-Bench are fully open-sourced, aiming to catalyze research on interactive, agentic, and general-purpose video intelligence for the next generation of multimodal AI systems. (https://univa.online/)
BibTeX
@misc{liang2025univa,
  author = {Zhengyang Liang and Daoan Zhang and Huichi Zhou and Rui Huang and Bobo Li and Yuechen Zhang and Shengqiong Wu and Xiaohan Wang and Jiebo Luo and Lizi Liao and Hao Fei},
  title = {{UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalist}},
  year = {2025},
  eprint = {2511.08521},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.08521},
}
Figure