TL;DR
CelebV-Text is a massive new open-source dataset featuring 70,000 high-quality facial video clips paired with 1.4 million detailed text descriptions. By capturing both static appearance details and dynamic, timestamped facial expressions and lighting changes, this dataset provides the rich multimodal data needed to train text-to-video models to generate highly realistic and text-accurate human animations.
What Problem Does This Paper Address?
While text-driven video generation has made massive strides recently, generating realistic human faces remains a notorious bottleneck. Existing models often produce blurry facial features or completely ignore parts of the text prompt.
The root of the problem lies in the data. Current face-centric video datasets are either too small, low-resolution, or lack descriptive text. Crucially, existing text-video datasets fail to capture temporal dynamics—how a face changes over time. A text prompt might ask for a person who "smiles and then frowns," but standard datasets don't provide the precise, timestamped text-video pairs required for a model to learn these transitions.
Methods
To bridge this gap, the authors developed a semi-automatic data construction pipeline to build CelebV-Text:
- Data Collection & Smooth Processing: The team collected high-resolution ($\ge 512 \times 512$) in-the-wild face videos. To ensure visual quality, they introduced a background-aware splitting strategy, cutting video clips whenever the background changed to eliminate jarring visual discontinuities.
- Hybrid Annotation Strategy: To balance cost and precision, they automated what they could and manually annotated the rest. General appearance, lighting conditions, and emotions were labeled using highly accurate automated models ($\ge 85\%$ accuracy). Human workers were brought in to define fine-grained static details (like the exact location of a mole or scar) and to provide precise start/end timestamps for dynamic actions (e.g., blinking, turning the head).
- Semi-Automatic Text Generation: Purely manual descriptions don't scale, but purely automated text sounds robotic. The authors analyzed human-written descriptions to find common grammar structures and used a Probabilistic Context-Free Grammar (PCFG) template-based approach. Combined with NLTK-powered synonym replacement, they generated 20 unique, natural-sounding, and highly complex text descriptions for every single video clip.
Datasets
The paper introduces CelebV-Text, which consists of:
- 70,000 video clips (totaling over 279 hours).
- 1,400,000 text descriptions (20 per clip).
The authors benchmarked CelebV-Text against existing face datasets, including MM-Vox (previously the only major facial text-video dataset, which is smaller and lacks dynamic descriptions) and CelebV-HQ (which has high-quality video and labels but completely lacks text descriptions).
Key Insights and Ideas
- Decoupling Static vs. Dynamic Attributes: Instead of treating a video caption as a single flat description, the authors explicitly separated static traits (e.g., "bushy eyebrows") from dynamic actions (e.g., "gazing, then laughing"). This structural separation allows models to better understand identity permanence versus motion.
- The Importance of Timestamps: By mapping text descriptions directly to exact video timestamps, the dataset teaches models when an action occurs, which is vital for generating coherent chronological video.
- Domain Specificity Beats Raw Model Size: A fascinating takeaway is that smaller baseline models trained specifically on CelebV-Text's high-quality data generated more text-relevant facial videos than massive, open-domain foundation models like CogVideo, which are up to 100x larger but lack fine-grained facial text training.
Key Results
- Superior Video Quality: Videos in CelebV-Text achieved significantly higher image and video quality scores (measured via BRISQUE and VSFA metrics) than MM-Vox, largely thanks to the background-aware clip splitting.
- Rich Text Complexity: The average text length in CelebV-Text is 67.15 words, compared to just 28.39 words for MM-Vox. It also shows massive linguistic diversity, utilizing 96 unique verbs compared to MM-Vox's 5.
- Stronger Text-Video Relevance: In text-to-video retrieval tasks (using the Clip2Video framework), CelebV-Text consistently outperformed alternative datasets across all recall metrics, proving that the generated text accurately reflects the video content.
Limitations and Future Work
- Legal and Privacy Constraints: Because the videos feature real people in-the-wild, the authors cannot distribute the raw video files directly due to copyright and privacy regulations. Instead, they provide the comprehensive annotations, processing tools, and download links.
- Future Directions: The authors plan to expand the dataset's scale and diversity. They also hope to leverage CelebV-Text for advanced tasks like fine-grained facial video editing, text-driven 3D facial generation, and exploring synthetic face generation frameworks to bypass the ethical hurdles inherent to real-world facial datasets.
Summary updated