TL;DR
"Faces that Speak" introduces a unified framework called TTSF that simultaneously generates natural speech and a synchronized talking video from just a single portrait and a text string. By disentangling facial motion from identity, the system achieves unprecedented voice consistency and realistic head movements, even for identities it has never seen during training.
What Problem Does This Paper Address?
Creating a "talking head" from text usually involves two separate steps: first, a Text-to-Speech (TTS) system generates audio, and then a Talking Face Generation (TFG) system animates a photo to match that audio. This "cascade" approach is clunky—it suffers from error accumulation, high computational latency, and often results in a "robotic" look where the head stays too still.
Furthermore, existing systems struggle with voice consistency. If you use a photo of someone to "style" a synthetic voice, the person's facial expression or head pose in that specific photo often bleeds into the audio characteristics. The authors set out to build a unified model that generates both modalities at once while ensuring that the generated voice sounds like the person in the photo, regardless of their pose.
Methods
The core of the paper is the Text-to-Speaking Face (TTSF) architecture. It integrates TFG and TTS into a single pipeline where features are shared and mutually beneficial.
- Identity Disentanglement: The model uses a visual encoder and a motion extractor to separate a portrait into two parts: an identity feature ($f_{id}$) and a motion feature ($f_m$). Crucially, the TTS system is conditioned only on the "motion-removed" identity feature, ensuring the voice remains consistent even if the input image has an extreme pose.
- OT-CFM Motion Sampler: To generate natural, varied head movements, the authors use Optimal-Transport Conditional Flow Matching (OT-CFM). Unlike standard diffusion models that can be slow, OT-CFM learns an efficient "flow" to transform random noise into realistic motion sequences.
- Motion Normalizer: To prevent the "shaky" artifacts common in generated videos, they include an auto-encoder-based noise reducer that compresses and cleans the motion features before they reach the generator.
- Audio Mapper: This module bridges the gap between text and lip movement. It uses a Multi-Receptive Field Fusion (MRF) module to combine linguistic data (text embeddings) with acoustic data (energy and upsampled features) to ensure the lips move precisely in sync with the generated sounds.
Datasets
The model was trained and evaluated on several prominent "in-the-wild" datasets:
- LRS3: The primary training set, consisting of thousands of spoken TED and TEDx videos.
- VoxCeleb2 & LRS2: Used for evaluation to test the model's ability to generalize to outdoor settings and unseen speakers.
Key Insights and Ideas
- Motion is Noise for TTS: One of the most clever insights is that facial motion in a source image acts as "noise" when trying to extract a speaker's voice style. By explicitly removing motion features ($f_s - f_m$), the authors produced significantly more consistent synthetic voices.
- Linguistic + Acoustic = Better Lips: The authors found that text alone isn't enough for perfect lip-sync. The Audio Mapper needs to see the "energy" of the speech to capture the intricate shapes of the mouth during specific sounds.
- Unified Training: Jointly training the TTS and TFG components allows the model to learn better-synchronized representations than if the two parts were trained in isolation.
Key Results
The TTSF framework outperformed state-of-the-art baselines like SadTalker and MakeItTalk across multiple metrics:
- Visual Quality: Achieved a superior FID score of 18.348 on the LRS2 dataset, indicating higher image realism.
- Identity Preservation: Recorded the highest ID-Similarity (0.864), meaning the generated video looks more like the original person than competing methods.
- Voice Similarity: In TTS evaluations, the model achieved a C-SIM of 0.593, significantly outperforming Face-TTS (0.272) in matching the target speaker's voice.
- Human Preference: In a user study, participants rated TTSF significantly higher for lip-sync accuracy, motion naturalness, and overall video realness.
Limitations and Future Work
While the system is robust, the authors note that the LSE-C (Lip Sync Error Confidence) metric, while standard, relies on a pre-trained model that may not always align perfectly with human perception.
Future work could involve expanding the model to handle even more extreme head rotations or integrating more granular emotional controls (e.g., making the generated face look "angry" or "happy" based on the text context). Additionally, while the system generalizes well to unseen identities, performance in extremely low-light or highly occluded "in-the-wild" scenarios remains a frontier for further refinement.
Summary updated