Abstract
Previous methods for audio-driven talking head generation assume the input audio to be clean with a neutral tone. As we show empirically, one can easily break these systems by simply adding certain background noise to the utterance or changing its emotional tone (to for example, sad). To make talking head generation robust to such variations, we propose an explicit audio representation learning framework that disentangles audio sequences into various factors such as phonetic content, emotional tone, background noise and others. We conduct experiments to validate that when conditioned on disentangled content representation, the generated mouth movement by our model is significantly more accurate than previous approaches (without disentangled learning) in the presence of noise and emotional variations. We further demonstrate that our framework is compatible with current state-of-the-art approaches by replacing their original component to learn audio based representation with ours. To the best of our knowledge, this is the first work which improves the performance of talking head generation through a disentangled audio representation perspective, which is important for many real-world applications.
Introduction
Most existing "talking head" models—AI that generates video of a person speaking based on an audio clip—rely on a major assumption: the input audio is clean and the speaker’s tone is neutral. In the real world, audio is rarely perfect. It contains background noise (wind, traffic, static) and emotional variations (shouting, crying, or whispering). When standard models encounter these variations, they often struggle to generalize. For example, a model might mistake a "sad" tone for a specific mouth shape or let background static distort the generated lip movements. Treating audio as a "black box" leads to overfitting on clean data and a lack of robustness in practical, "in-the-wild" applications.
Methodology: Learning Disentangled Representations
The authors propose a two-stage approach centered on a Variational Autoencoder (VAE) to explicitly factorize audio into three distinct latent spaces: Phonetic Content ($z_c$), Emotional Tone ($z_e$), and Sequence-level variations ($z_s$, capturing noise and speaker-specific traits).
To ensure these factors are truly separate, the framework employs a Margin Ranking Loss. This forces the content representation to be excellent at predicting mouth shapes (visemes) but intentionally poor at predicting emotion or noise, and vice versa. This "lateral disentanglement" prevents the video generator from getting confused by non-phonetic audio cues. Once disentangled, only the content representation ($z_c$) is fed into a GAN-based video generator consisting of a U-Net architecture and dual discriminators (frame-level and video-level) to ensure temporal smoothness.
Key Results and Robustness
The model was evaluated on the GRID, CREMA-D, and LRS3 datasets. Using Landmark Distance (LMD) to measure accuracy, the proposed model demonstrated superior stability. In moderate noise (-30dB), the model maintained an LMD of 1.33, while the previous state-of-the-art (Vougioukas et al.) jumped to 1.96.
Qualitatively, the model produced consistent mouth movements across different emotional tones. In contrast, competing models often produced abrupt or "weak" mouth movements when the speaker was sad or happy. A major insight is the modular compatibility of this approach; the authors demonstrated that replacing the audio components of existing SOTA models with their disentangled version immediately made those models more robust to noise.
Conclusion
By explicitly forcing the model to categorize noise and emotion into separate latent "buckets," the authors prove that disentanglement significantly improves lip-sync accuracy. While performance eventually degrades under extreme noise where phonetic information is lost, this framework provides a robust foundation for real-world talking head applications. Future work may extend this to disentangling identity-specific factors or applying these representations to 3D mesh animation.
Summary updated