TL;DR
Current text-to-video (T2V) models often fail when asked to generate scenes that real users actually care about, primarily because training sets recycle open-domain footage rather than addressing user intent. VideoUFO fills this gap with 1.09 million CC-BY-licensed video clips spanning 1,291 user-focused topics extracted directly from real-world prompt logs. Fine-tuning a baseline generator on VideoUFO closes the performance gap on difficult prompts, significantly boosting visual fidelity on previously neglected concepts.
What Problem Does This Paper Address?
State-of-the-art generative video models (such as Sora) can render impressive cinematic landscapes, yet they routinely stumble on specific concepts requested by users—such as generating a biologically plausible glowing firefly rather than just a dark meadow.
The authors identify two root causes in data curation:
- Topic Mismatch: Most video datasets are curated from indiscriminate web scrapes that miss the long tail of subjects users actually request.
- Data Inbreeding: Many recent datasets (e.g., Panda-70M, OpenVid-1M, VidGen-1M) do not contribute novel footage; they simply re-filter and re-caption subsets of HD-VILA-100M under restrictive research-only licenses. In fact, combining all popular existing datasets still covers less than 80% of common user-requested topics.
Methods
The authors designed a targeted data curation pipeline directly steered by user demand:
- User Topic Mining: The team embedded 1.67 million real user prompts from VidProM using SentenceTransformers, clustered them with $K$-means ($K=2,000$), and prompted GPT-4o to extract 1–2 word topic descriptors. After deduplication and pruning overly broad terms (like "film" or "movement"), they settled on 1,291 user-focused topics.
- Targeted YouTube Harvesting: Using the official YouTube API, they collected ~500 videos per topic under Creative Commons licenses ($\ge 720\text{p}$, $<4$ minutes), gathering 586,490 source videos. Crucially, only 0.29% overlapped with existing datasets.
- Clip Splitting & Dense Annotation: Footage was segmented into semantic shots (averaging 12.6 seconds). Each clip was paired with both a brief caption (via Panda-70M) and a dense caption (via Qwen2-VL-7B).
- Frugal Clip Verification: To ensure clips actually depicted the queried topic without spending thousands on multimodal API calls, the team fed the dense text captions into GPT-4o-mini. This proxy verification achieved ~80.6% agreement with full-video GPT-4o inspection while slashing API costs by over 5,400× ($0.03 vs. $162 per 1,000 clips), yielding 1,091,712 verified clips.
- Quality Filtering: Each clip is scored across six VBench dimensions (e.g., subject consistency, motion smoothness, aesthetic quality).
VidProM Prompts (1.67M)
│ (K-Means + GPT-4o)
▼
1,291 User Topics ──► YouTube API (CC-BY, ≥720p) ──► 586K Raw Videos
│ (Shot Segmentation)
▼
3.18M Candidate Clips
│ (Dense Captioning)
▼
GPT-4o-mini Verification
│ (Topic Filtering)
▼
1.09M VideoUFO Clips
Datasets
- VideoUFO (Introduced): 1.09M video clips across 1,291 user topics, released under a permissive CC BY 4.0 license.
- BenchUFO (Introduced): An evaluation benchmark comprising 791 concrete noun topics, each tested with 10 real-world user prompts (7,910 total prompts). To score generation accuracy, generated videos are re-captioned with Qwen2-VL-7B, and the cosine similarity between the generated description and the input prompt is computed.
- VidProM: Utilized as the source pool of 1.67M user prompts.
- Comparison Baselines: Compared against nine existing video corpora, including HD-VILA-100M, WebVid-10M, InternVid, Panda-70M, LVD-2M, MiraData, Koala-36M, VidGen-1M, and OpenVid-1M.
Key Insights and Ideas
- Datasets Have Blind Spots: Off-the-shelf T2V models show consistent high quality on animal categories (e.g., seagulls, pandas, dolphins) due to internet data biases, but catastrophically fail on human-centric, scientific, or niche artistic requests (e.g., giant squid, ancient Egyptian, animal cell, Van Gogh).
- Existing Coverage is Saturated: Even an aggregate union of all major public video datasets leaves more than 20% of user-focused topics completely unrepresented.
- Dense Text is an Effective Video Proxy: Evaluating detailed visual captions with small language models provides a scalable alternative to running heavy vision-language models across millions of raw video files for filtering.
Key Results
Evaluating 16 public and open-source models (including HunyuanVideo, CogVideoX, LTX-Video, and Mochi-1) on BenchUFO revealed a steep drop between best-performing and worst-performing topics (similarity gaps between Top-10 and Low-10 topics ranged from 0.233 to 0.314).
BenchUFO Low-10 Topic Similarity (Higher is Better)
──────────────────────────────────────────────────────────
Pyramidal [0.400]
MVDiT-VidGen [0.382]
MVDiT-OpenVid [0.401]
MVDiT-VideoUFO (Ours) [0.442] (+4.2% over prior SOTA)
──────────────────────────────────────────────────────────
- When training an MVDiT baseline exclusively on VideoUFO, its score on the 10 worst-performing topics jumped from 0.401 to 0.442 (+4.2% absolute gain), outperforming models trained on similar-scale datasets like OpenVid-1M and VidGen-1M.
- Top-performing topic accuracy was preserved (0.651 vs. 0.645 on OpenVid-1M).
- Qualitative evaluations demonstrated clear wins on difficult prompts, successfully rendering intricate historical attire, microscopic cells, and niche fantasy hybrids where competing models output distorted artifacts.
Limitations and Future Work
- Compute Restraints: Due to academic GPU limitations (tested on 32 A100s), the authors evaluated VideoUFO on moderate-sized baselines (MVDiT) rather than multi-billion-parameter foundation models.
- Benchmark Scope: BenchUFO focuses strictly on concrete nouns. Abstract concepts (e.g., "freedom" or "love") were excluded because multimodal models summarize the metaphorical visuals (like doves or hearts), causing artificial prompt-description embedding mismatches.
- Future Directions: Expanding the pipeline from text-to-video into image-to-video (I2V) using datasets like TIP-I2V, and scaling the collection to 10M+ clips as prompt distributions evolve.
Summary updated