VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models

Wenhao Wang, Yi Yang

The "Sora era" of text-to-video (T2V) generation has arrived, promising a future where cinematic masterpieces are just a prompt away. But while we marvel at the output, we know surprisingly little about the input—the complex, often messy prompts that real users write to coax these models into action. Enter VidProM, a massive new dataset that aims to be the "DiffusionDB of video," providing the first large-scale look at how humans actually interact with T2V models.

TL;DR

VidProM is the first million-scale dataset of real-world text-to-video prompts, featuring 1.67 million unique prompts from real users and 6.69 million videos generated by four state-of-the-art models. It reveals that video prompts are fundamentally different from image prompts—longer, more dynamic, and obsessed with time—and provides a vital benchmark for research into prompt engineering, fake video detection, and copyright.

What Problem Does This Paper Address?

In the world of generative AI, "garbage in, garbage out" is the golden rule. To build better models, we need to understand the prompts users actually write. While the text-to-image (T2I) community has DiffusionDB, the T2V community has largely relied on datasets of "captions" (descriptions of existing real-world videos).

The problem? Captions aren't prompts. A caption describes what is in a video; a prompt is a set of instructions meant to guide a model. Furthermore, video generation introduces a "time dimension" that image generation lacks. Until now, there was no public, large-scale dataset that captured the nuances of how users describe motion, duration, and temporal transitions.

Methods

The authors curated VidProM through a rigorous pipeline:

  1. Data Collection: They scraped over 10 official Pika Discord channels between July 2023 and February 2024, capturing real-time user interactions.
  2. Cleaning & Filtering: They removed image-to-video prompts and failed generations, resulting in 1.67 million unique T2V prompts.
  3. Embedding & Safety: Each prompt was embedded using OpenAI’s text-embedding-3-large (3072 dimensions) and screened for "Not-Safe-For-Work" (NSFW) content using the Detoxify model, assigning probabilities for toxicity, obscenity, and more.
  4. Multi-Model Generation: To make the dataset robust, they didn't just keep the original Pika videos. They used the prompts to generate new videos using three additional open-source models: VideoCraft2, Text2Video-Zero, and ModelScope.
  5. Semantic De-duplication: They introduced VidProS, a subset of 1.04 million "semantically unique" prompts (where no two prompts have a cosine similarity > 0.8), ensuring high diversity for training.
An example entry from VidProM showing the prompt, metadata, safety scores, and video frames generated by four different models.

Datasets

  • VidProM: The primary dataset containing 1.67M unique prompts and 6.69M generated videos.
  • VidProS: A curated subset of 1.04M semantically unique prompts.
  • Comparison Benchmarks: The paper compares VidProM against DiffusionDB (the T2I standard) and Panda-70M (a standard video-caption dataset) to highlight the unique characteristics of T2V prompts.

Key Insights and Ideas

  • The "Time Dimension" Gap: T2V prompts are significantly more complex than T2I prompts. They frequently include descriptions of actions ("flying," "working"), transitions, and specific durations ("1-minute realistic short video").
  • Prompt Length: T2V users are wordy. VidProM contains nearly 60,000 prompts longer than 70 words, whereas DiffusionDB has almost none. This suggests that guiding motion requires more descriptive "scaffolding" than static scenes.
Comparison of prompts from DiffusionDB (text-to-image) and VidProM (text-to-video), highlighting the significantly longer word counts and distinct semantic clustering of video prompts.
  • The Domain Gap: There is a massive disconnect between the captions used to train models (like "a person walking") and the prompts users actually want to run (like "Spider-Man in a cyberpunk city"). This gap likely hinders the performance of current models on creative, stylized content.
  • Copyright & Replication: The authors found that T2V models occasionally "memorize" and replicate training data (e.g., generating a near-perfect copy of Dalí’s The Persistence of Memory), raising significant legal and ethical questions.

Key Results

  • Scale: 1.67 million unique prompts and 6.69 million videos.
  • Diversity: VidProM contains 40.6% more semantically unique prompts than DiffusionDB, despite having a similar total count, indicating a broader range of topics.
  • Fake Video Detection Benchmark: The authors tested eight state-of-the-art fake image detectors on VidProM. The results were mediocre, with average accuracies hovering around 50-60%, proving that current "deepfake" detectors are ill-equipped for the nuances of diffusion-generated video.
  • Efficiency: By using the semantic de-duplication algorithm, the authors were able to process the entire million-scale dataset in just 0.604 hours using 8 A100 GPUs.

Limitations and Future Work

The authors are transparent about several hurdles:

  • Video Quality: The videos generated by the four models used (Pika, ModelScope, etc.) are relatively short and lower resolution compared to the latest closed-source models like Sora.
  • The "Curse of Recursion": As more AI-generated videos populate the web, there is a risk of models being trained on their own synthetic output, which can lead to "model collapse" or irreversible defects.
  • Future Expansion: The team plans to incorporate videos from more advanced models (like Sora or CogVideoX) as they become accessible, and they suggest that VidProM could be used to train "automatic prompt completion" models to help users write better instructions.

VidProM is now publicly available under a CC-BY-NC 4.0 license at https://vidprom.github.io.

Summary updated

Abstract

The arrival of Sora marks a new era for text-to-video diffusion models, bringing significant advancements in video generation and potential applications. However, Sora, along with other text-to-video diffusion models, is highly reliant on prompts, and there is no publicly available dataset that features a study of text-to-video prompts. In this paper, we introduce VidProM, the first large-scale dataset comprising 1.67 Million unique text-to-Video Prompts from real users. Additionally, this dataset includes 6.69 million videos generated by four state-of-the-art diffusion models, alongside some related data. We initially discuss the curation of this large-scale dataset, a process that is both time-consuming and costly. Subsequently, we underscore the need for a new prompt dataset specifically designed for text-to-video generation by illustrating how VidProM differs from DiffusionDB, a large-scale prompt-gallery dataset for image generation. Our extensive and diverse dataset also opens up many exciting new research areas. For instance, we suggest exploring text-to-video prompt engineering, efficient video generation, and video copy detection for diffusion models to develop better, more efficient, and safer models. The project (including the collected dataset VidProM and related code) is publicly available at https://vidprom.github.io under the CC-BY-NC 4.0 License.
BibTeX
@inproceedings{wang2024vidprom,
  author = {Wenhao Wang and Yi Yang},
  title = {{VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure