Abstract: All previous methods for audio-driven talking head generation assume the input audio to be clean with a neutral tone. As we show empirically, one can easily break these systems by simply adding certain background noise to the utterance or changing its emotional tone (to such as sad). To make talking head generation robust to such variations, we propose an explicit audio representation learning framework that disentangles audio sequences into various factors such as phonetic content, emotional tone, background noise and others. We conduct experiments to validate that conditioned on disentangled content representation, the generated mouth movement by our model is significantly more accurate than previous approaches (without disentangled learning) in the presence of noise and emotional variations. We further demonstrate that our framework is compatible with current state-of-the-art approaches by replacing their original audio learning component with ours. To our best knowledge, this is the first work which improves the performance of talking head generation from disentangled audio representation perspective, which is important for many real-world applications.
Introduction
Generating a "talking head" video from a single image and an audio clip is a classic challenge in computer vision. While existing models perform well with studio-quality, neutral speech, they often fall apart in real-world conditions. Researchers from Microsoft have developed a novel framework that remains robust even when the input audio is noisy or emotionally charged. By using a Variational Autoencoder (VAE) to explicitly disentangle phonetic content from emotional tone and background noise, the system produces significantly more accurate lip synchronization than previous state-of-the-art models.
The Problem: Noise and Emotion
The authors identify two main "deal-breakers" for current systems:
- Background Noise: Even moderate ambient noise can confuse a model, leading to jittery or frozen mouth movements.
- Emotional Variation: A person saying the same word while crying versus laughing produces different acoustic signals. Most models struggle to find the underlying "phonetic" truth beneath these emotional layers.
Previous methods treated audio as a "black box," hoping the neural network would implicitly learn to ignore noise. This paper argues for explicit disentanglement: separating the "what" (speech content) from the "how" (emotion and noise) before trying to animate the face.
Methodology
The proposed framework utilizes a two-stage approach to ensure robustness.
1. Learning Disentangled Representations
The core of the system is a Variational Autoencoder (VAE) designed to factorize the audio into three distinct latent spaces:
- Content ($Z_c$): Captures the phonetic information (the "visemes").
- Emotion ($Z_e$): Captures the emotional tone (happy, sad, etc.).
- Sequence ($Z_s$): Captures residual variations like background noise.
To ensure these factors do not "leak" into each other, the authors introduce a Margin Ranking Loss. This forces the content encoder to perform well at predicting speech sounds but intentionally poorly at predicting emotion or noise, effectively "filtering" the representation.
2. Talking Head Generation
The animation is handled by a Generative Adversarial Network (GAN). The generator takes a source identity image and the disentangled content representation ($Z_c$) to synthesize video frames. Because the generator only "sees" the cleaned-up content, it isn't distracted by the emotional or noisy artifacts in the original signal.
Experimental Setup
The study utilizes three key datasets to train and validate the model:
- GRID: A high-quality audiovisual corpus of 34 talkers, used for its phonetic diversity.
- CREMA-D: Contains over 7,000 clips of actors speaking with six different emotions, crucial for training emotion disentanglement.
- LRS3 (Lip Reading Sentences 3): An "in-the-wild" dataset from TED videos used to test generalization to unconstrained environments.
Key Results
The researchers evaluated their model using Landmark Distance (LMD), which measures the Euclidean distance between the generated lip positions and the ground truth.
- Noise Robustness: At -30dB noise, the proposed model maintained an LMD of ~1.33 on the GRID dataset, while previous state-of-the-art (SOTA) models saw their error jump significantly (e.g., from 1.3 to 1.96).
- Emotional Consistency: Qualitatively, the model produced consistent mouth movements for the same sentence regardless of whether the input audio was "Happy" or "Sad." In contrast, baseline models often produced "weak" or "abrupt" mouth movements when faced with non-neutral emotions.
- Plug-and-Play Compatibility: The disentangled audio encoder can be "swapped" into existing SOTA architectures (like DAVS), immediately improving their robustness to noise without retraining the entire visual generator.
Conclusion
This work demonstrates that explicit disentanglement is superior to implicit learning for robust talking head generation. While the model is highly resilient, it faces limitations at extremely high noise levels (e.g., -10dB) where the underlying phonetic signal is too distorted to recover. Future work may extend this disentanglement to other factors, such as speaker identity and gender, for even finer animation control.
Reference: Mittal, G., & Wang, B. (2019). Animating Face using Disentangled Audio Representations. arXiv:1910.00726v1 [cs.CV].
Summary updated