Abstract
Text-to-video (T2V) generative models convert textual prompts into dynamic visual content, offering wide-ranging applications in film production, gaming, and education. However, their real-world performance often falls short of user expectations. One key reason is that these models have not been trained on videos related to some topics users want to create. In this paper, we propose VideoUFO, the first video dataset specifically curated to align with Users' FOcus in real-world scenarios. Beyond this, our VideoUFO also features: (1) minimal (0.29%) overlap with existing video datasets, and (2) videos searched exclusively via YouTube's official API under the Creative Commons license. These two attributes provide future researchers with greater freedom to broaden their training sources. The VideoUFO comprises over 1.09 million video clips, each paired with both a brief and a detailed caption. Specifically, through clustering, we first identify 1,291 user-focused topics from the million-scale real text-to-video prompt dataset, VidProM. Then, we use these topics to retrieve videos from YouTube, split the retrieved videos into clips, and generate both brief and detailed captions for each clip. After verifying the clips with specified topics, we are left with about 1.09 million video clips. Our experiments reveal that (1) current text-to-video models do not achieve consistent performance across all user-focused topics; and (2) a simple model trained on VideoUFO outperforms others on worst-performing topics.
Introduction: The Data Distribution Problem
Despite the rapid advancement of text-to-video (T2V) models, a significant gap remains between model capabilities and user expectations. State-of-the-art models often struggle with specific, niche prompts—for example, successfully rendering a "night" scene but failing to accurately depict a "glowing firefly." This discrepancy stems from a fundamental data distribution problem: most existing T2V datasets are scraped from open-domain sources that do not necessarily reflect the creative interests of actual users. Furthermore, many datasets suffer from "knowledge stagnation" due to repetitive re-processing of older data and lack clear copyright compliance.
Methodology: The VideoUFO Pipeline
To address these issues, we developed VideoUFO (where FO stands for Users' FOcus), a million-scale dataset created through a rigorous five-step pipeline:
- Topic Extraction: We analyzed 1.67 million real-world prompts from the VidProM dataset, using SentenceTransformers and K-means clustering to identify 2,000 groups, which GPT-4o then summarized into 1,291 distinct, human-verified topics.
- Targeted Retrieval: Videos matching these topics were retrieved via the official YouTube API, specifically filtering for Creative Commons licenses to ensure legal usability.
- Processing & Segmentation: Long videos were split into semantically consistent short clips using shot boundary detection.
- Dual-Level Captioning: Every clip received both a brief caption and a highly detailed description generated by QWen2-VL-7B.
- Cost-Effective Verification: To ensure topic alignment, we utilized GPT-4o mini to verify the detailed text captions rather than the raw video clips, reducing API costs by over 5,000x.
Dataset Characteristics
VideoUFO consists of 1.09 million video clips across 1,291 topics. Notably, it has only a 0.29% overlap with existing major datasets like HD-VILA-100M or Panda-70M. By sourcing fresh content directly from YouTube, VideoUFO introduces novel knowledge that has not yet been integrated into current generative models.
Performance Evaluation and Insights
Our research uncovered a significant "blind spot" in current T2V models: massive performance gaps between their best and worst topics. While models excel at common subjects like animals, they frequently fail on specific cultural or technical concepts.
To evaluate this, we introduced BenchUFO, a specialized benchmark of 791 concrete noun topics. In our experiments, a standard MVDiT model trained on VideoUFO achieved a "Low 10" score (the average of the 10 worst-performing topics) of 0.442, significantly outperforming versions trained on OpenVid (0.401) or VidGen (0.382). Qualitative results show that VideoUFO-trained models are superior at following complex, multi-concept prompts.
Conclusion
VideoUFO demonstrates that aligning training data with user intent is more effective than simply increasing data volume. By focusing on the 1,291 topics users actually care about, we can significantly boost model performance on difficult subjects. Future work will explore scaling VideoUFO to 100 million clips and extending this user-focused methodology to the image-to-video domain.
Summary updated