The "Sora era" of text-to-video (T2V) generation has arrived, promising a future where cinematic masterpieces are just a prompt away. But while we marvel at the output, we know surprisingly little about the input—the complex, often messy prompts that real users write to coax these models into action. Enter VidProM, a massive new dataset that aims to be the "DiffusionDB of video," providing the first large-scale look at how humans actually interact with T2V models.
TL;DR
VidProM is the first million-scale dataset of real-world text-to-video prompts, featuring 1.67 million unique prompts from real users and 6.69 million videos generated by four state-of-the-art models. It reveals that video prompts are fundamentally different from image prompts—longer, more dynamic, and obsessed with time—and provides a vital benchmark for research into prompt engineering, fake video detection, and copyright.
What Problem Does This Paper Address?
In the world of generative AI, "garbage in, garbage out" is the golden rule. To build better models, we need to understand the prompts users actually write. While the text-to-image (T2I) community has DiffusionDB, the T2V community has largely relied on datasets of "captions" (descriptions of existing real-world videos).
The problem? Captions aren't prompts. A caption describes what is in a video; a prompt is a set of instructions meant to guide a model. Furthermore, video generation introduces a "time dimension" that image generation lacks. Until now, there was no public, large-scale dataset that captured the nuances of how users describe motion, duration, and temporal transitions.
Methods
The authors curated VidProM through a rigorous pipeline:
- Data Collection: They scraped over 10 official Pika Discord channels between July 2023 and February 2024, capturing real-time user interactions.
- Cleaning & Filtering: They removed image-to-video prompts and failed generations, resulting in 1.67 million unique T2V prompts.
- Embedding & Safety: Each prompt was embedded using OpenAI’s
text-embedding-3-large(3072 dimensions) and screened for "Not-Safe-For-Work" (NSFW) content using the Detoxify model, assigning probabilities for toxicity, obscenity, and more. - Multi-Model Generation: To make the dataset robust, they didn't just keep the original Pika videos. They used the prompts to generate new videos using three additional open-source models: VideoCraft2, Text2Video-Zero, and ModelScope.
- Semantic De-duplication: They introduced VidProS, a subset of 1.04 million "semantically unique" prompts (where no two prompts have a cosine similarity > 0.8), ensuring high diversity for training.
Datasets
- VidProM: The primary dataset containing 1.67M unique prompts and 6.69M generated videos.
- VidProS: A curated subset of 1.04M semantically unique prompts.
- Comparison Benchmarks: The paper compares VidProM against DiffusionDB (the T2I standard) and Panda-70M (a standard video-caption dataset) to highlight the unique characteristics of T2V prompts.
Key Insights and Ideas
- The "Time Dimension" Gap: T2V prompts are significantly more complex than T2I prompts. They frequently include descriptions of actions ("flying," "working"), transitions, and specific durations ("1-minute realistic short video").
- Prompt Length: T2V users are wordy. VidProM contains nearly 60,000 prompts longer than 70 words, whereas DiffusionDB has almost none. This suggests that guiding motion requires more descriptive "scaffolding" than static scenes.
- The Domain Gap: There is a massive disconnect between the captions used to train models (like "a person walking") and the prompts users actually want to run (like "Spider-Man in a cyberpunk city"). This gap likely hinders the performance of current models on creative, stylized content.
- Copyright & Replication: The authors found that T2V models occasionally "memorize" and replicate training data (e.g., generating a near-perfect copy of Dalí’s The Persistence of Memory), raising significant legal and ethical questions.
Key Results
- Scale: 1.67 million unique prompts and 6.69 million videos.
- Diversity: VidProM contains 40.6% more semantically unique prompts than DiffusionDB, despite having a similar total count, indicating a broader range of topics.
- Fake Video Detection Benchmark: The authors tested eight state-of-the-art fake image detectors on VidProM. The results were mediocre, with average accuracies hovering around 50-60%, proving that current "deepfake" detectors are ill-equipped for the nuances of diffusion-generated video.
- Efficiency: By using the semantic de-duplication algorithm, the authors were able to process the entire million-scale dataset in just 0.604 hours using 8 A100 GPUs.
Limitations and Future Work
The authors are transparent about several hurdles:
- Video Quality: The videos generated by the four models used (Pika, ModelScope, etc.) are relatively short and lower resolution compared to the latest closed-source models like Sora.
- The "Curse of Recursion": As more AI-generated videos populate the web, there is a risk of models being trained on their own synthetic output, which can lead to "model collapse" or irreversible defects.
- Future Expansion: The team plans to incorporate videos from more advanced models (like Sora or CogVideoX) as they become accessible, and they suggest that VidProM could be used to train "automatic prompt completion" models to help users write better instructions.
VidProM is now publicly available under a CC-BY-NC 4.0 license at https://vidprom.github.io.
Summary updated