Faces that Speak: Jointly Synthesising Talking Face and Speech from Text

Youngjoon Jang, Ji-Hoon Kim, Junseok Ahn, Doyeop Kwak, Hong-Sun Yang, Yoon-Cheol Ju, Il-Hwan Kim, Byeong-Yeol Kim, Joon Son Chung

TL;DR

"Faces that Speak" introduces a unified framework called TTSF that simultaneously generates natural speech and a synchronized talking video from just a single portrait and a text string. By disentangling facial motion from identity, the system achieves unprecedented voice consistency and realistic head movements, even for identities it has never seen during training.

What Problem Does This Paper Address?

Creating a "talking head" from text usually involves two separate steps: first, a Text-to-Speech (TTS) system generates audio, and then a Talking Face Generation (TFG) system animates a photo to match that audio. This "cascade" approach is clunky—it suffers from error accumulation, high computational latency, and often results in a "robotic" look where the head stays too still.

Furthermore, existing systems struggle with voice consistency. If you use a photo of someone to "style" a synthetic voice, the person's facial expression or head pose in that specific photo often bleeds into the audio characteristics. The authors set out to build a unified model that generates both modalities at once while ensuring that the generated voice sounds like the person in the photo, regardless of their pose.

Figure 1. Our framework integrates Talking Face Generation (TFG) and Text-to-Speech (TTS) systems, generating synchronised natural speech and a talking face video from a single portrait and text input. Our model is capable of variational motion generation by conditioning the TFG model with the intermediate representations of the TTS model. The speech is conditioned using the identity features extracted in the TFG model to align with the input identity.

Methods

The core of the paper is the Text-to-Speaking Face (TTSF) architecture. It integrates TFG and TTS into a single pipeline where features are shared and mutually beneficial.

  1. Identity Disentanglement: The model uses a visual encoder and a motion extractor to separate a portrait into two parts: an identity feature ($f_{id}$) and a motion feature ($f_m$). Crucially, the TTS system is conditioned only on the "motion-removed" identity feature, ensuring the voice remains consistent even if the input image has an extreme pose.
  2. OT-CFM Motion Sampler: To generate natural, varied head movements, the authors use Optimal-Transport Conditional Flow Matching (OT-CFM). Unlike standard diffusion models that can be slow, OT-CFM learns an efficient "flow" to transform random noise into realistic motion sequences.
  3. Motion Normalizer: To prevent the "shaky" artifacts common in generated videos, they include an auto-encoder-based noise reducer that compresses and cleans the motion features before they reach the generator.
  4. Audio Mapper: This module bridges the gap between text and lip movement. It uses a Multi-Receptive Field Fusion (MRF) module to combine linguistic data (text embeddings) with acoustic data (energy and upsampled features) to ensure the lips move precisely in sync with the generated sounds.
Figure 2. Overall architecture of our framework. The TTS model receives identity representations from the TFG model, while the TFG model takes conditions for natural motion generation from the TTS model. These complementary elements enhance our model’s capabilities in generating both speech and talking faces. The EMB block denotes an embedding operation. The grey dashed arrow represents a path used only during the training process, and the red arrows represent paths used only during the inference process.

Datasets

The model was trained and evaluated on several prominent "in-the-wild" datasets:

  • LRS3: The primary training set, consisting of thousands of spoken TED and TEDx videos.
  • VoxCeleb2 & LRS2: Used for evaluation to test the model's ability to generalize to outdoor settings and unseen speakers.

Key Insights and Ideas

  • Motion is Noise for TTS: One of the most clever insights is that facial motion in a source image acts as "noise" when trying to extract a speaker's voice style. By explicitly removing motion features ($f_s - f_m$), the authors produced significantly more consistent synthetic voices.
  • Linguistic + Acoustic = Better Lips: The authors found that text alone isn't enough for perfect lip-sync. The Audio Mapper needs to see the "energy" of the speech to capture the intricate shapes of the mouth during specific sounds.
  • Unified Training: Jointly training the TTS and TFG components allows the model to learn better-synchronized representations than if the two parts were trained in isolation.
Figure 3. The architecture of the audio mapper. The condition denotes the concatenated feature of text embedding et, upsampled text feature ˜ft, and energy, which is a norm of ˜ft.

Key Results

The TTSF framework outperformed state-of-the-art baselines like SadTalker and MakeItTalk across multiple metrics:

  • Visual Quality: Achieved a superior FID score of 18.348 on the LRS2 dataset, indicating higher image realism.
  • Identity Preservation: Recorded the highest ID-Similarity (0.864), meaning the generated video looks more like the original person than competing methods.
  • Voice Similarity: In TTS evaluations, the model achieved a C-SIM of 0.593, significantly outperforming Face-TTS (0.272) in matching the target speaker's voice.
  • Human Preference: In a user study, participants rated TTSF significantly higher for lip-sync accuracy, motion naturalness, and overall video realness.

Limitations and Future Work

While the system is robust, the authors note that the LSE-C (Lip Sync Error Confidence) metric, while standard, relies on a pre-trained model that may not always align perfectly with human perception.

Future work could involve expanding the model to handle even more extreme head rotations or integrating more granular emotional controls (e.g., making the generated face look "angry" or "happy" based on the text context). Additionally, while the system generalizes well to unseen identities, performance in extremely low-light or highly occluded "in-the-wild" scenarios remains a frontier for further refinement.

Summary updated

Abstract

The goal of this work is to simultaneously generate natural talking faces and speech outputs from text. We achieve this by integrating Talking Face Generation (TFG) and Text-to-Speech (TTS) systems into a unified framework. We address the main challenges of each task: (1) generating a range of head poses representative of real-world scenarios, and (2) ensuring voice consistency despite variations in facial motion for the same identity. To tackle these issues, we introduce a motion sampler based on conditional flow matching, which is capable of high-quality motion code generation in an efficient way. Moreover, we introduce a novel conditioning method for the TTS system, which utilises motion-removed features from the TFG model to yield uniform speech outputs. Our extensive experiments demonstrate that our method effectively creates natural-looking talking faces and speech that accurately match the input text. To our knowledge, this is the first effort to build a multimodal synthesis system that can generalise to unseen identities.
BibTeX
@inproceedings{jang2024faces,
  author = {Youngjoon Jang and Ji-Hoon Kim and Junseok Ahn and Doyeop Kwak and Hong-Sun Yang and Yoon-Cheol Ju and Il-Hwan Kim and Byeong-Yeol Kim and Joon Son Chung},
  title = {{Faces that Speak: Jointly Synthesising Talking Face and Speech from Text}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=NrFCBB3V79},
}
Figure