Talking Faces: Audio-to-Video Face Generation

Yuxin Wang, Linsen Song, Wayne Wu, Chen Qian, Ran He, Chen Change Loy, Chen Qian 0006, Ran He 0001Handbook of Digital Face Manipulation and Detection 2022 · DOI · DBLP

Abstract

This paper provides a comprehensive survey of "talking face generation"—the AI-driven process of synthesizing realistic facial videos from audio inputs. It maps the evolution of the field from simple 1990s animations to modern deep-learning models capable of generating photo-realistic, emotionally expressive videos from just a single image and a snippet of speech.

1. Introduction

The core challenge in this field is cross-modal synthesis: mapping a dynamic, 1D audio signal (speech) to a coherent, high-dimensional 2D or 3D video sequence. To overcome the "Uncanny Valley," researchers must solve sub-problems including precise lip synchronization, fine-grained control of non-verbal cues (eye blinks, head poses), and generalization to unseen identities without extensive training data.

2. Methodology

The standard talking face pipeline consists of four distinct stages: audio representation, face modeling, audio-to-face animation, and post-processing.

2.1 Audio Representation

Instead of raw waveforms, modern models utilize robust features like Mel-frequency cepstrum coefficients (MFCC) or high-level text features (phonemes). Recent trends involve using pre-trained models such as DeepSpeech to extract noise-resistant content features.

Common audio representations used in synthesis: (a) waveform, (b) spectrogram, (c) phonetic transcription, and (d) feature map.

2.2 Face Modeling and Animation

Face modeling employs either 2D landmarks/Action Units or 3D Morphable Face Models (3DMM) for structural stability. The animation "bridge" has evolved from early Hidden Markov Models (HMMs) to state-of-the-art Generative Adversarial Networks (GANs) and Encoder-Decoder architectures. These models are designed to disentangle "identity" from "speech content."

3. Datasets

Research relies on two main categories of data:

  • Indoor (Controlled): Datasets like GRID and TCD-TIMIT provide high-quality, frontal-view videos. The MEAD dataset is notable for its focus on emotional diversity across multiple intensities.
  • In-the-Wild (Unconstrained): Real-world clips from platforms like YouTube, including VoxCeleb1 & 2 and LRS3, offer thousands of speakers and sentences for robust training.

4. Key Insights and Results

  • The Return to 3D: The field is shifting back toward 3D representations, which provide a geometric prior that facilitates handling large head movements and maintaining facial consistency.
  • Disentanglement: Successful models treat talking faces as a combination of independent factors: identity, pose, expression, and speech. This allows for cross-identity synthesis (e.g., making Person A speak with Person B's voice).
  • Efficiency Gains: High-fidelity synthesis has progressed from requiring 14 hours of specific footage to "one-shot" or "few-shot" models that generate convincing results from a single static image.

5. Limitations and Future Work

Despite progress, rendering the interior of the mouth (teeth and tongue) remains difficult. Additionally, fine-grained control over emotional intensity in 2D models is an ongoing challenge. Finally, the rise of deepfakes necessitates significant investment in detection technologies and ethical AI frameworks to prevent fraud and misinformation.

Summary updated

BibTeX
@inproceedings{wang2022talking,
  author = {Yuxin Wang and Linsen Song and Wayne Wu and Chen Qian and Ran He and Chen Change Loy and Chen Qian and Ran He},
  title = {{Talking Faces: Audio-to-Video Face Generation}},
  booktitle = {Handbook of Digital Face Manipulation and Detection},
  year = {2022},
  doi = {10.1007/978-3-030-87664-7\_8},
}
Figure