VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video Generation

Wenhao Wang, Yi Yang

TL;DR

Current text-to-video (T2V) models often fail when asked to generate scenes that real users actually care about, primarily because training sets recycle open-domain footage rather than addressing user intent. VideoUFO fills this gap with 1.09 million CC-BY-licensed video clips spanning 1,291 user-focused topics extracted directly from real-world prompt logs. Fine-tuning a baseline generator on VideoUFO closes the performance gap on difficult prompts, significantly boosting visual fidelity on previously neglected concepts.

What Problem Does This Paper Address?

State-of-the-art generative video models (such as Sora) can render impressive cinematic landscapes, yet they routinely stumble on specific concepts requested by users—such as generating a biologically plausible glowing firefly rather than just a dark meadow.

The authors identify two root causes in data curation:

  1. Topic Mismatch: Most video datasets are curated from indiscriminate web scrapes that miss the long tail of subjects users actually request.
  2. Data Inbreeding: Many recent datasets (e.g., Panda-70M, OpenVid-1M, VidGen-1M) do not contribute novel footage; they simply re-filter and re-caption subsets of HD-VILA-100M under restrictive research-only licenses. In fact, combining all popular existing datasets still covers less than 80% of common user-requested topics.

Methods

The authors designed a targeted data curation pipeline directly steered by user demand:

  1. User Topic Mining: The team embedded 1.67 million real user prompts from VidProM using SentenceTransformers, clustered them with $K$-means ($K=2,000$), and prompted GPT-4o to extract 1–2 word topic descriptors. After deduplication and pruning overly broad terms (like "film" or "movement"), they settled on 1,291 user-focused topics.
  2. Targeted YouTube Harvesting: Using the official YouTube API, they collected ~500 videos per topic under Creative Commons licenses ($\ge 720\text{p}$, $<4$ minutes), gathering 586,490 source videos. Crucially, only 0.29% overlapped with existing datasets.
  3. Clip Splitting & Dense Annotation: Footage was segmented into semantic shots (averaging 12.6 seconds). Each clip was paired with both a brief caption (via Panda-70M) and a dense caption (via Qwen2-VL-7B).
  4. Frugal Clip Verification: To ensure clips actually depicted the queried topic without spending thousands on multimodal API calls, the team fed the dense text captions into GPT-4o-mini. This proxy verification achieved ~80.6% agreement with full-video GPT-4o inspection while slashing API costs by over 5,400× ($0.03 vs. $162 per 1,000 clips), yielding 1,091,712 verified clips.
  5. Quality Filtering: Each clip is scored across six VBench dimensions (e.g., subject consistency, motion smoothness, aesthetic quality).
VidProM Prompts (1.67M)
   │ (K-Means + GPT-4o)
   ▼
1,291 User Topics ──► YouTube API (CC-BY, ≥720p) ──► 586K Raw Videos
                                                            │ (Shot Segmentation)
                                                            ▼
                                                     3.18M Candidate Clips
                                                            │ (Dense Captioning)
                                                            ▼
                                                     GPT-4o-mini Verification
                                                            │ (Topic Filtering)
                                                            ▼
                                                     1.09M VideoUFO Clips

Datasets

  • VideoUFO (Introduced): 1.09M video clips across 1,291 user topics, released under a permissive CC BY 4.0 license.
  • BenchUFO (Introduced): An evaluation benchmark comprising 791 concrete noun topics, each tested with 10 real-world user prompts (7,910 total prompts). To score generation accuracy, generated videos are re-captioned with Qwen2-VL-7B, and the cosine similarity between the generated description and the input prompt is computed.
  • VidProM: Utilized as the source pool of 1.67M user prompts.
  • Comparison Baselines: Compared against nine existing video corpora, including HD-VILA-100M, WebVid-10M, InternVid, Panda-70M, LVD-2M, MiraData, Koala-36M, VidGen-1M, and OpenVid-1M.

Key Insights and Ideas

  • Datasets Have Blind Spots: Off-the-shelf T2V models show consistent high quality on animal categories (e.g., seagulls, pandas, dolphins) due to internet data biases, but catastrophically fail on human-centric, scientific, or niche artistic requests (e.g., giant squid, ancient Egyptian, animal cell, Van Gogh).
  • Existing Coverage is Saturated: Even an aggregate union of all major public video datasets leaves more than 20% of user-focused topics completely unrepresented.
  • Dense Text is an Effective Video Proxy: Evaluating detailed visual captions with small language models provides a scalable alternative to running heavy vision-language models across millions of raw video files for filtering.

Key Results

Evaluating 16 public and open-source models (including HunyuanVideo, CogVideoX, LTX-Video, and Mochi-1) on BenchUFO revealed a steep drop between best-performing and worst-performing topics (similarity gaps between Top-10 and Low-10 topics ranged from 0.233 to 0.314).

BenchUFO Low-10 Topic Similarity (Higher is Better)
──────────────────────────────────────────────────────────
Pyramidal             [0.400]
MVDiT-VidGen          [0.382]
MVDiT-OpenVid         [0.401]
MVDiT-VideoUFO (Ours) [0.442]  (+4.2% over prior SOTA)
──────────────────────────────────────────────────────────
  • When training an MVDiT baseline exclusively on VideoUFO, its score on the 10 worst-performing topics jumped from 0.401 to 0.442 (+4.2% absolute gain), outperforming models trained on similar-scale datasets like OpenVid-1M and VidGen-1M.
  • Top-performing topic accuracy was preserved (0.651 vs. 0.645 on OpenVid-1M).
  • Qualitative evaluations demonstrated clear wins on difficult prompts, successfully rendering intricate historical attire, microscopic cells, and niche fantasy hybrids where competing models output distorted artifacts.

Limitations and Future Work

  • Compute Restraints: Due to academic GPU limitations (tested on 32 A100s), the authors evaluated VideoUFO on moderate-sized baselines (MVDiT) rather than multi-billion-parameter foundation models.
  • Benchmark Scope: BenchUFO focuses strictly on concrete nouns. Abstract concepts (e.g., "freedom" or "love") were excluded because multimodal models summarize the metaphorical visuals (like doves or hearts), causing artificial prompt-description embedding mismatches.
  • Future Directions: Expanding the pipeline from text-to-video into image-to-video (I2V) using datasets like TIP-I2V, and scaling the collection to 10M+ clips as prompt distributions evolve.

Summary updated

Abstract

Text-to-video generative models convert textual prompts into dynamic visual content, offering wide-ranging applications in film production, gaming, and education. However, their real-world performance often falls short of user expectations. One key reason is that these models have not been trained on videos related to some topics users want to create. In this paper, we propose VideoUFO, the first Video dataset specifically curated to align with Users' FOcus in real-world scenarios. Beyond this, our VideoUFO also features: (1) minimal (0.29\%) overlap with existing video datasets, and (2) videos searched exclusively via YouTube's official API under the Creative Commons license. These two attributes provide future researchers with greater freedom to broaden their training sources. The VideoUFO comprises over 1.09 million video clips, each paired with both a brief and a detailed caption (description). Specifically, through clustering, we first identify 1,291 user-focused topics from the million-scale real text-to-video prompt dataset, VidProM. Then, we use these topics to retrieve videos from YouTube, split the retrieved videos into clips, and generate both brief and detailed captions for each clip. After verifying the clips with specified topics, we are left with about 1.09 million video clips. Our experiments reveal that (1) current 16 text-to-video models do not achieve consistent performance across all user-focused topics; and (2) a simple model trained on VideoUFO outperforms others on worst-performing topics. The dataset and code are publicly available at https://huggingface.co/datasets/WenhaoWang/VideoUFO and https://github.com/WangWenhao0716/BenchUFO under the CC BY 4.0 License.
BibTeX
@inproceedings{wang2025videoufo,
  author = {Wenhao Wang and Yi Yang},
  title = {{VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video Generation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=wwlwRuKle7},
}

Loading related content...

Figure