VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models

Wenhao Wang, Yi Yang

TL;DR

The paper introduces VidProM, the first large-scale dataset of over 1.67 million real-world text-to-video prompts and 6.69 million corresponding generated videos. This dataset aims to address the critical need for prompt-centric research in the rapidly advancing field of text-to-video diffusion models, paving the way for improved model development, prompt engineering, and safety.

What Problem Does This Paper Address?

The recent surge in text-to-video diffusion models, exemplified by OpenAI's Sora, has revolutionized video generation. However, these powerful models are heavily reliant on the quality and specificity of user prompts. Despite this crucial dependency, there's a significant lack of publicly available datasets specifically designed for studying text-to-video prompts. Existing prompt datasets are primarily for image generation, and their prompts often differ in complexity and dynamism from those needed for video. This gap hinders research into prompt engineering, model evaluation, and the development of more robust and nuanced text-to-video systems.

Methods

The core of this work is the creation of the VidProM dataset. The authors collected prompts from official Pika Discord channels, filtering out image-to-video prompts and those without associated videos. They then generated videos using four state-of-the-art diffusion models: Pika, VideoCraft2, Text2Video-Zero, and ModelScope. To ensure data quality and enable further analysis, each prompt was embedded using OpenAI's text-embedding-3-large model, and NSFW probabilities were assigned using Detoxify. A subset of semantically unique prompts (VidProS) was also created by filtering prompts with cosine similarity below 0.8. The paper also details a semantic de-duplication algorithm used to achieve this.

Datasets

  • VidProM: The primary contribution, a new dataset comprising:
  • 1.67 million unique text-to-video prompts from real users.
  • 6.69 million generated videos from four diffusion models (Pika, VideoCraft2, Text2Video-Zero, ModelScope).
  • Associated metadata including UUIDs, timestamps, NSFW probabilities, and 3072-dimensional prompt embeddings.
  • VidProS: A subset of VidProM containing semantically unique prompts.
  • DiffusionDB: A large-scale prompt-gallery dataset for image generation, used as a point of comparison to highlight the unique characteristics of VidProM.
  • Panda-70M: Used in conjunction with VidProM to illustrate the domain gap between video captions and real user prompts.

Key Insights and Ideas

  • Prompts for Video are Different: Text-to-video prompts are inherently more dynamic, complex, and longer than text-to-image prompts, often requiring descriptions of actions, transitions, and durations.
  • The Need for Specialized Datasets: The significant semantic differences between image and video prompts, as visualized by WizMap, underscore the necessity of dedicated datasets like VidProM for advancing text-to-video research.
  • Beyond Simple Uniqueness: The concept of "semantically unique prompts" is introduced, emphasizing the importance of diverse topics and themes in a dataset for broader coverage.
  • Current Fake Video Detection is Limited: Existing fake image detection methods struggle to generalize to fake videos generated by diffusion models, highlighting the need for specialized video detection techniques.
  • Replication is a Concern: Text-to-video models can replicate content from training or existing videos, raising potential copyright issues if misused.

Key Results

  • VidProM contains 1,672,243 unique prompts and 6,690,000 generated videos.
  • VidProM prompts are significantly longer than those in DiffusionDB, with nearly 60,000 prompts having over 70 words.
  • The dataset includes NSFW probabilities for six categories, with a very small fraction of prompts exceeding a 0.2 threshold.
  • The creation of VidProM required approximately 50,631 GPU hours.
  • Benchmarking fake video detection methods on VidProM showed that traditional methods focusing on global texture and frequency analysis were most effective, while specialized diffusion-based methods struggled.
  • Video copy detection analysis revealed that while replication is present, it's a small proportion, with commercial models showing lower rates than open-source ones.

Limitations and Future Work

  • Video Quality and Length: The generated videos are noted to be short and not of the highest quality. Future work aims to incorporate higher-quality videos from more advanced models like Sora.
  • Fake Video Detection Generalization: Current fake video detection models do not generalize well to diffusion-generated videos, necessitating the development of new, specialized models trained on datasets like VidProM.
  • Video Copy Detection: Existing video copy detection models, trained on manual transformations, fail to detect diffusion-generated replications. Future work suggests using these models as filters and then employing human labelers to build a new detection model.
  • Prompt Engineering: VidProM provides a rich resource for developing automatic text-to-video prompt completion models and exploring prompt engineering techniques.
  • Efficient Video Generation: The dataset can be used to develop methods for efficient video generation by searching for similar prompts and reconstructing videos, rather than generating from scratch.

Summary updated

Abstract

The arrival of Sora marks a new era for text-to-video diffusion models, bringing significant advancements in video generation and potential applications. However, Sora, along with other text-to-video diffusion models, is highly reliant on prompts, and there is no publicly available dataset that features a study of text-to-video prompts. In this paper, we introduce VidProM, the first large-scale dataset comprising 1.67 Million unique text-to-Video Prompts from real users. Additionally, this dataset includes 6.69 million videos generated by four state-of-the-art diffusion models, alongside some related data. We initially discuss the curation of this large-scale dataset, a process that is both time-consuming and costly. Subsequently, we underscore the need for a new prompt dataset specifically designed for text-to-video generation by illustrating how VidProM differs from DiffusionDB, a large-scale prompt-gallery dataset for image generation. Our extensive and diverse dataset also opens up many exciting new research areas. For instance, we suggest exploring text-to-video prompt engineering, efficient video generation, and video copy detection for diffusion models to develop better, more efficient, and safer models. The project (including the collected dataset VidProM and related code) is publicly available at https://vidprom.github.io under the CC-BY-NC 4.0 License.
BibTeX
@inproceedings{wang2024vidprom,
  author = {Wenhao Wang and Yi Yang},
  title = {{VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}

Loading related content...

Figure