Introduction
UniVA is an open-source, multi-agent framework that unifies video understanding, generation, editing, and segmentation into a single, cohesive workflow. By employing a "Plan-and-Act" architecture and a hierarchical memory system, it enables complex, multi-step video creation—like generating a scene and then iteratively editing specific objects—that previously required manual coordination of multiple disparate AI tools.
The Challenge of Composite Video Workflows
While the AI field has seen an explosion of specialized models for video (e.g., models that only generate or only segment), real-world production requires composite workflows. For instance, a creator might want to generate a video from text, swap a character's outfit, and then extend the clip—all while maintaining visual consistency.
Current solutions fall into two camps, both with significant drawbacks:
- Single-task models: These are high-performing but isolated; they cannot reason across multiple steps without human "glue."
- Monolithic Video-Language Models (VLMs): These are often inflexible and difficult to extend with new specialized tools.
The authors identify a critical bottleneck: the lack of a unified system capable of long-horizon reasoning, proactive assistance, and seamless tool orchestration for video intelligence.
System Architecture: The Plan-and-Act Paradigm
UniVA introduces a dual-agent system designed for "Agentic Synergy." Its architecture is built on three pillars:
1. Plan-and-Act Architecture
- Planner Agent: Interprets high-level user intent and decomposes it into a structured sequence of subtasks.
- Executor Agent: Carries out these steps by invoking modular tools via the Model Context Protocol (MCP). This allows UniVA to be "omni-capable," easily plugging in new state-of-the-art models as tool servers.
2. Hierarchical Multi-level Memory
To sustain long-term context, UniVA uses three memory layers:
- Global Memory: Stores persistent knowledge and historical traces to improve future planning.
- Task Memory: Maintains intermediate artifacts (like segmentation masks or captions) so they can be reused across steps without recomputation.
- User Memory: Tracks specific user preferences and styles for personalized interaction.
3. Self-Reflective Loops
The system doesn't just execute; it evaluates. If a generated video doesn't meet the requirements, the agent can initiate self-reflection to course-correct and adapt its plan. Because the system is agentic and modular, every step of the video production process is traceable, allowing for "any-conditioned" workflows (text-to-video, image-to-video, or video-to-video).
UniVA-Bench: Evaluating Agentic Intelligence
The authors introduce UniVA-Bench, a new benchmark suite designed to evaluate agentic video intelligence rather than just single-task accuracy. It leverages existing datasets like Video-MME (for understanding), Opens2v-nexus (for generation), and DAVIS2017 (for segmentation), but reformulates them into multi-step "goal cards."
Beyond standard quality scores (CLIP, DINO), it introduces Agentic Metrics:
- Weighted Plan Edit Distance (wPED): Measures the accuracy of the generated plan against a ground truth.
- Dependency Coverage (DepCov): Evaluates how well the agent handles tasks that depend on previous outputs.
- Re-planning Quality (ReplanQ): Measures the agent's ability to recover from execution failures.
Key Insights and Results
A major insight from this work is the integration of perception and reasoning. UniVA uses its understanding module to help its perception tools. For example, if a "blue car" is occluded in a video, the agent can use narrative context to confirm it's the same car when it reappears, a task that pixel-level segmentation models often struggle with.
UniVA demonstrates superior performance across the board compared to both specialized models and general-purpose LLMs:
- Understanding: On the LongVideo Understanding task, UniVA achieved an accuracy of 0.76, outperforming GPT-4o (0.52) and Gemini 2.5 Pro (0.65).
- Segmentation: It significantly outperformed the SA2VA baseline, achieving a J&F score of 0.2467 vs. 0.1524.
- Planning Robustness: The Plan-Act framework more than doubled the "Success Rate" (producing a structurally valid plan) compared to a single-agent setup (45.0% vs. 20.0%).
- Human Preference: In a formal study, UniVA was the clear leader, winning in 4 out of 5 dimensions, including Semantic Accuracy and Style Consistency.
Conclusion
UniVA represents a significant step toward a universal video generalist. While it excels at complex instructions and narrative coherence, the authors acknowledge a current trade-off: it sometimes sacrifices strict frame-level similarity (DINO score) compared to specialized, non-agentic models. Future work will focus on optimizing the balance between high-level narrative control and fine-grained visual consistency, expanding the open-source tool ecosystem, and improving multi-agent communication efficiency.
Summary updated