Abstract
UniVid demonstrates that a video generation model, pre-trained solely on natural videos, can be transformed into a "universal visual backbone" for nearly any vision task. By representing tasks as "visual sentences" (e.g., Image A → Mask A' :: Image B → Mask B') and applying lightweight fine-tuning, the researchers show that these models can seamlessly handle both image and video understanding and generation across mixed modalities.
Introduction
Current Large Vision Models (LVMs) attempt to mimic the success of Large Language Models (LLMs) by treating visual data as sequences. However, they face a "fragmented data" bottleneck: to learn diverse tasks like segmentation or depth estimation, they typically require massive, labor-intensive pre-training on specific, annotated datasets.
Furthermore, existing LVMs are often "modality-locked"—they struggle to generalize when a prompt uses an image to describe a task for a video query (cross-modal inference). They also lack a unified way to switch between understanding (e.g., generating a mask from a video) and generation (e.g., generating a video from a mask). UniVid aims to solve these scalability and flexibility issues by leveraging the inherent temporal reasoning of pre-trained video diffusion models.
Methodology
The authors propose UniVid, a framework built on a pre-trained video Diffusion Transformer (specifically, the Wan2.1-T2V-1.3B model).
Visual Sentence Paradigm
Tasks are structured as a sequence $V = [A, A', B, B']$. In this paradigm, $A \to A'$ serves as the "context" or example (e.g., a natural video and its corresponding edge map), $B$ is the query, and $B'$ is the target the model must predict. Because video models are designed to process frames over time, UniVid treats these four components as segments of a single video. The model's 3D full-attention mechanism naturally captures the relationships between the example pair and the query.
Architectural Adaptation
Instead of massive retraining, the authors use Supervised Fine-Tuning (SFT) with Low-Rank Adaptation (LoRA). This allows the model to learn new tasks (like semantic segmentation) with as few as 20 training samples. By simply swapping the order of the visual sentence (e.g., $A' \to A$), the model switches from an "understanding" task to a "generation" task without any architectural changes.
Key Insights
- Video Models as Natural Sequence Learners: Pre-training on natural videos teaches a model temporal dependencies that translate well to the "logical logic" of a task prompt (If $A$ becomes $A'$, then $B$ should become $B'$).
- Cross-Modal Fluidity: Unlike previous models, UniVid can handle "mixed" contexts. For example, it can use a static image and its depth map as an example to successfully predict the depth map for a moving video query.
- The Symmetry of Vision: Understanding and generation are treated as mathematically similar. Reordering the visual sentence is the only "switch" needed to turn a segmentation model into a video synthesis model.
- Co-training Synergy: Joint multi-task fine-tuning yielded better results than training on each task individually, suggesting the model learns a generalized "meta-task" of following visual instructions.
Key Results
UniVid significantly outperformed the baseline LVM [1] across multiple benchmarks:
- Depth Estimation: On the NYU-v2 dataset, UniVid achieved an RMSE of 2.55, compared to LVM's 74.16 (lower is better).
- Semantic Segmentation: On ADE20K, UniVid reached a mean Intersection over Union (mIoU) of 8.712, vastly outperforming LVM's 1.423.
- Few-Shot Efficiency: The model demonstrated strong performance with as few as 20 samples per task, proving that the pre-trained video backbone already contains most of the necessary "visual intelligence."
Limitations and Future Work
The current backbone is limited to 81 frames per sequence, restricting the duration of clips in the visual sentence. Additionally, because the model uses a generative diffusion process, it can occasionally produce "hallucinated" or inconsistent labels in dense prediction tasks. Future directions include exploring longer-context video models and investigating ways to reduce the inherent randomness of diffusion for more precise outputs.
Summary updated