Abstract
Unified video modeling that combines generation and understanding capabilities is increasingly important but faces two key challenges: maintaining semantic faithfulness during flow-based generation due to text-visual token imbalance and the limitations of uniform cross-modal attention across the flow trajectory, and efficiently extending image-centric MLLMs to video without costly retraining. We present UniVid, a unified architecture that couples an MLLM with a diffusion decoder through a lightweight adapter, enabling both video understanding and generation. We introduce Temperature Modality Alignment to improve prompt adherence and Pyramid Reflection for efficient temporal reasoning via dynamic keyframe selection. Extensive experiments on standard benchmarks demonstrate state-of-the-art performance, achieving a 2.2% improvement on VBench-Long total score compared to EasyAnimateV5.1, and 1.0% and 3.3% accuracy gains on MSVD-QA and ActivityNet-QA, respectively, compared with the best prior 7B baselines.
Introduction
In the current artificial intelligence landscape, video models typically diverge into two specialized domains: generative models (e.g., Sora, Kling) focused on visual synthesis, and understanding models (Video-LLMs) focused on descriptive analysis. Bridging these capabilities within a single framework remains a significant challenge. Generative models often suffer from "semantic drift," where the high volume of visual tokens dilutes the influence of textual prompts. Conversely, understanding models struggle with temporal efficiency, as uniform frame sampling often misses critical evidence required for complex reasoning. UniVid addresses these hurdles by unifying generation and understanding through a shared multimodal large language model (MLLM) core.
Unified Architecture
UniVid adopts a hybrid architecture that leverages a frozen MLLM as a central reasoning engine, coupled with a Diffusion Transformer (DiT) decoder (specifically Wan 2.2) for video synthesis. This coupling is achieved via a lightweight adapter, allowing the model to perform both tasks without the prohibitive cost of training a massive end-to-end transformer from scratch.
Technical Innovations
To enhance performance in both domains, UniVid introduces two primary methodological innovations:
1. Temperature Modality Alignment (TMA)
TMA is a timestep-aware scheduling mechanism designed for the generation process. It addresses the imbalance between textual and visual tokens by dynamically adjusting guidance. In the early stages of diffusion, TMA strengthens textual guidance to establish the scene's layout and semantic structure. As the process progresses, it shifts focus toward visual refinement to ensure high-quality pixel synthesis, effectively preventing semantic drift.
2. Pyramid Reflection
For video understanding, UniVid employs a "test-time reinforcement learning" loop known as Pyramid Reflection. This Actor-Evaluator-Reflector cycle enables the model to reason iteratively. If the Evaluator assigns a low confidence score to an initial answer, the Reflector suggests a refined search query. This allows the model to dynamically expand or prune its evidence set (keyframes) until it reaches a high-confidence conclusion, mimicking human-like temporal reasoning.
Experimental Results
UniVid was evaluated on a mix of high-quality open-source datasets, including OpenVid-1M for generation and ActivityNet-QA for understanding.
Generation Performance
On the VBench-Long benchmark, UniVid achieved a state-of-the-art total score of 85.27, representing a 2.2% improvement over EasyAnimateV5.1. The model demonstrates superior semantic fidelity and object consistency, particularly in maintaining complex motion trajectories over long horizons.
Understanding Performance
In the understanding domain, UniVid set new records for 7B-scale models, achieving 80.1% accuracy on MSVD-QA and 58.8% on ActivityNet-QA. These results highlight the effectiveness of the Pyramid Reflection mechanism in identifying salient temporal information.
Conclusion
UniVid represents a significant step toward unified video intelligence. By combining a reasoning-heavy MLLM with a high-fidelity diffusion decoder, it achieves state-of-the-art results in both generation and understanding. While the current architecture introduces some computational overhead, future work will focus on integrating native video encoders and 3D VAEs to further improve efficiency and handle more complex temporal dynamics.
Summary updated