CelebV-Text: A Large-Scale Facial Text-Video Dataset

Jianhui Yu, Hao Zhu, Liming Jiang, Chen Change Loy, Weidong Cai, Wayne Wu

TL;DR

CelebV-Text is a massive new open-source dataset featuring 70,000 high-quality facial video clips paired with 1.4 million detailed text descriptions. By capturing both static appearance details and dynamic, timestamped facial expressions and lighting changes, this dataset provides the rich multimodal data needed to train text-to-video models to generate highly realistic and text-accurate human animations.

What Problem Does This Paper Address?

While text-driven video generation has made massive strides recently, generating realistic human faces remains a notorious bottleneck. Existing models often produce blurry facial features or completely ignore parts of the text prompt.

The root of the problem lies in the data. Current face-centric video datasets are either too small, low-resolution, or lack descriptive text. Crucially, existing text-video datasets fail to capture temporal dynamics—how a face changes over time. A text prompt might ask for a person who "smiles and then frowns," but standard datasets don't provide the precise, timestamped text-video pairs required for a model to learn these transitions.

Methods

To bridge this gap, the authors developed a semi-automatic data construction pipeline to build CelebV-Text:

  1. Data Collection & Smooth Processing: The team collected high-resolution ($\ge 512 \times 512$) in-the-wild face videos. To ensure visual quality, they introduced a background-aware splitting strategy, cutting video clips whenever the background changed to eliminate jarring visual discontinuities.
  2. Hybrid Annotation Strategy: To balance cost and precision, they automated what they could and manually annotated the rest. General appearance, lighting conditions, and emotions were labeled using highly accurate automated models ($\ge 85\%$ accuracy). Human workers were brought in to define fine-grained static details (like the exact location of a mole or scar) and to provide precise start/end timestamps for dynamic actions (e.g., blinking, turning the head).
  3. Semi-Automatic Text Generation: Purely manual descriptions don't scale, but purely automated text sounds robotic. The authors analyzed human-written descriptions to find common grammar structures and used a Probabilistic Context-Free Grammar (PCFG) template-based approach. Combined with NLTK-powered synonym replacement, they generated 20 unique, natural-sounding, and highly complex text descriptions for every single video clip.
Figure 2. Pipeline of our dataset construction process. The pipeline includes data collection & processing, data annotation, and semi-auto text generation.

Datasets

The paper introduces CelebV-Text, which consists of:

  • 70,000 video clips (totaling over 279 hours).
  • 1,400,000 text descriptions (20 per clip).

The authors benchmarked CelebV-Text against existing face datasets, including MM-Vox (previously the only major facial text-video dataset, which is smaller and lacks dynamic descriptions) and CelebV-HQ (which has high-quality video and labels but completely lacks text descriptions).

Key Insights and Ideas

  • Decoupling Static vs. Dynamic Attributes: Instead of treating a video caption as a single flat description, the authors explicitly separated static traits (e.g., "bushy eyebrows") from dynamic actions (e.g., "gazing, then laughing"). This structural separation allows models to better understand identity permanence versus motion.
  • The Importance of Timestamps: By mapping text descriptions directly to exact video timestamps, the dataset teaches models when an action occurs, which is vital for generating coherent chronological video.
  • Domain Specificity Beats Raw Model Size: A fascinating takeaway is that smaller baseline models trained specifically on CelebV-Text's high-quality data generated more text-relevant facial videos than massive, open-domain foundation models like CogVideo, which are up to 100x larger but lack fine-grained facial text training.

Key Results

  • Superior Video Quality: Videos in CelebV-Text achieved significantly higher image and video quality scores (measured via BRISQUE and VSFA metrics) than MM-Vox, largely thanks to the background-aware clip splitting.
Figure 4. Dataset quality distribution. The metrics used are BRISQUE [44] and VSFA [34] respectively.
  • Rich Text Complexity: The average text length in CelebV-Text is 67.15 words, compared to just 28.39 words for MM-Vox. It also shows massive linguistic diversity, utilizing 96 unique verbs compared to MM-Vox's 5.
  • Stronger Text-Video Relevance: In text-to-video retrieval tasks (using the Clip2Video framework), CelebV-Text consistently outperformed alternative datasets across all recall metrics, proving that the generated text accurately reflects the video content.

Limitations and Future Work

  • Legal and Privacy Constraints: Because the videos feature real people in-the-wild, the authors cannot distribute the raw video files directly due to copyright and privacy regulations. Instead, they provide the comprehensive annotations, processing tools, and download links.
  • Future Directions: The authors plan to expand the dataset's scale and diversity. They also hope to leverage CelebV-Text for advanced tasks like fine-grained facial video editing, text-driven 3D facial generation, and exploring synthetic face generation frameworks to bypass the ethical hurdles inherent to real-world facial datasets.

Summary updated

Abstract

Text-driven generation models are flourishing in video generation and editing. However, face-centric text-to-video generation remains a challenge due to the lack of a suitable dataset containing high-quality videos and highly relevant texts. This paper presents CelebV-Text, a large-scale, diverse, and high-quality dataset of facial text-video pairs, to facilitate research on facial text-to-video generation tasks. CelebV-Text comprises 70,000 in-the-wild face video clips with diverse visual content, each paired with 20 texts generated using the proposed semi-automatic text generation strategy. The provided texts are of high quality, describing both static and dynamic attributes precisely. The superiority of CelebV-Text over other datasets is demonstrated via comprehensive statistical analysis of the videos, texts, and text-video relevance. The effectiveness and potential of CelebV-Text are further shown through extensive self-evaluation. A benchmark is constructed with representative methods to standardize the evaluation of the facial text-to-video generation task. All data and models are publicly available.
BibTeX
@inproceedings{yu2023celebv,
  author = {Jianhui Yu and Hao Zhu and Liming Jiang and Chen Change Loy and Weidong Cai and Wayne Wu},
  title = {{CelebV-Text: A Large-Scale Facial Text-Video Dataset}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure