VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video Generation

Wenhao Wang, Yi Yang

Abstract

Text-to-video (T2V) generative models convert textual prompts into dynamic visual content, offering wide-ranging applications in film production, gaming, and education. However, their real-world performance often falls short of user expectations. One key reason is that these models have not been trained on videos related to some topics users want to create. In this paper, we propose VideoUFO, the first video dataset specifically curated to align with Users' FOcus in real-world scenarios. Beyond this, our VideoUFO also features: (1) minimal (0.29%) overlap with existing video datasets, and (2) videos searched exclusively via YouTube's official API under the Creative Commons license. These two attributes provide future researchers with greater freedom to broaden their training sources. The VideoUFO comprises over 1.09 million video clips, each paired with both a brief and a detailed caption. Specifically, through clustering, we first identify 1,291 user-focused topics from the million-scale real text-to-video prompt dataset, VidProM. Then, we use these topics to retrieve videos from YouTube, split the retrieved videos into clips, and generate both brief and detailed captions for each clip. After verifying the clips with specified topics, we are left with about 1.09 million video clips. Our experiments reveal that (1) current text-to-video models do not achieve consistent performance across all user-focused topics; and (2) a simple model trained on VideoUFO outperforms others on worst-performing topics.

Introduction: The Data Distribution Problem

Despite the rapid advancement of text-to-video (T2V) models, a significant gap remains between model capabilities and user expectations. State-of-the-art models often struggle with specific, niche prompts—for example, successfully rendering a "night" scene but failing to accurately depict a "glowing firefly." This discrepancy stems from a fundamental data distribution problem: most existing T2V datasets are scraped from open-domain sources that do not necessarily reflect the creative interests of actual users. Furthermore, many datasets suffer from "knowledge stagnation" due to repetitive re-processing of older data and lack clear copyright compliance.

Methodology: The VideoUFO Pipeline

To address these issues, we developed VideoUFO (where FO stands for Users' FOcus), a million-scale dataset created through a rigorous five-step pipeline:

  1. Topic Extraction: We analyzed 1.67 million real-world prompts from the VidProM dataset, using SentenceTransformers and K-means clustering to identify 2,000 groups, which GPT-4o then summarized into 1,291 distinct, human-verified topics.
  2. Targeted Retrieval: Videos matching these topics were retrieved via the official YouTube API, specifically filtering for Creative Commons licenses to ensure legal usability.
  3. Processing & Segmentation: Long videos were split into semantically consistent short clips using shot boundary detection.
  4. Dual-Level Captioning: Every clip received both a brief caption and a highly detailed description generated by QWen2-VL-7B.
  5. Cost-Effective Verification: To ensure topic alignment, we utilized GPT-4o mini to verify the detailed text captions rather than the raw video clips, reducing API costs by over 5,000x.

Dataset Characteristics

VideoUFO consists of 1.09 million video clips across 1,291 topics. Notably, it has only a 0.29% overlap with existing major datasets like HD-VILA-100M or Panda-70M. By sourcing fresh content directly from YouTube, VideoUFO introduces novel knowledge that has not yet been integrated into current generative models.

Figure 5: High-quality animal-themed content, such as this kitten on a beach, represents topics where current models often excel but still benefit from high-fidelity training data.

Performance Evaluation and Insights

Our research uncovered a significant "blind spot" in current T2V models: massive performance gaps between their best and worst topics. While models excel at common subjects like animals, they frequently fail on specific cultural or technical concepts.

To evaluate this, we introduced BenchUFO, a specialized benchmark of 791 concrete noun topics. In our experiments, a standard MVDiT model trained on VideoUFO achieved a "Low 10" score (the average of the 10 worst-performing topics) of 0.442, significantly outperforming versions trained on OpenVid (0.401) or VidGen (0.382). Qualitative results show that VideoUFO-trained models are superior at following complex, multi-concept prompts.

Figure 6: A serene bamboo water feature illustrates the type of specific, detailed content VideoUFO provides to help models bridge the gap in niche user-focused topics.

Conclusion

VideoUFO demonstrates that aligning training data with user intent is more effective than simply increasing data volume. By focusing on the 1,291 topics users actually care about, we can significantly boost model performance on difficult subjects. Future work will explore scaling VideoUFO to 100 million clips and extending this user-focused methodology to the image-to-video domain.

Summary updated

Abstract

Text-to-video generative models convert textual prompts into dynamic visual content, offering wide-ranging applications in film production, gaming, and education. However, their real-world performance often falls short of user expectations. One key reason is that these models have not been trained on videos related to some topics users want to create. In this paper, we propose VideoUFO, the first Video dataset specifically curated to align with Users' FOcus in real-world scenarios. Beyond this, our VideoUFO also features: (1) minimal (0.29%) overlap with existing video datasets, and (2) videos searched exclusively via YouTube's official API under the Creative Commons license. These two attributes provide future researchers with greater freedom to broaden their training sources. The VideoUFO comprises over 1.09 million video clips, each paired with both a brief and a detailed caption (description). Specifically, through clustering, we first identify 1,291 user-focused topics from the million-scale real text-to-video prompt dataset, VidProM. Then, we use these topics to retrieve videos from YouTube, split the retrieved videos into clips, and generate both brief and detailed captions for each clip. After verifying the clips with specified topics, we are left with about 1.09 million video clips. Our experiments reveal that (1) current 16 text-to-video models do not achieve consistent performance across all user-focused topics; and (2) a simple model trained on VideoUFO outperforms others on worst-performing topics. The dataset and code are publicly available at https://huggingface.co/datasets/WenhaoWang/VideoUFO and https://github.com/WangWenhao0716/BenchUFO under the CC BY 4.0 License.
BibTeX
@inproceedings{wang2025videoufo,
  author = {Wenhao Wang and Yi Yang},
  title = {{VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video Generation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=wwlwRuKle7},
}

Loading related content...

Figure