Memory-Augmented Contrastive Learning for Talking Head Generation

Jianrong Wang, Yaxin Zhao, Hongkai Fan, Tianyi Xu, Qi Li, Sen Li, Li Liu

Abstract

Generating realistic talking head videos from a single reference image and an audio clip remains a significant challenge due to the stochastic nature of human speech and movement. This paper presents a novel framework that addresses the "one-to-many" mapping problem—where a single audio segment can naturally correspond to multiple head poses and expressions. By integrating a memory-augmented contrastive learning approach for speech feature extraction with Mixture Density Networks (MDN) for landmark prediction, the proposed model generates synchronized, rhythmic, and natural head movements that surpass current state-of-the-art methods in both accuracy and visual quality.

Introduction

The synthesis of talking head videos is a classic problem in computer vision and signal processing. The primary difficulty lies in the "uncertainty gap": standard supervised learning models often utilize Mean Squared Error (MSE) loss, which encourages the model to find a single "average" output. In the context of human movement, this results in "blurry" or robotic animations because the model is effectively averaging out all possible valid movements for a given sound. Furthermore, extracting high-quality speech features typically requires vast amounts of labeled data, which is often noisy or expensive to acquire.

This research introduces a three-stage pipeline designed to embrace the inherent uncertainty of human expression rather than smoothing it over. By decoupling speech content from physical variability and modeling the latter as a probability distribution, the framework achieves high-fidelity lip synchronization and natural head dynamics.

Methodology

1. Memory-Augmented Speech Feature Extractor (SFE)

To move beyond the limitations of traditional supervised learning, the framework employs self-supervised contrastive learning. The Speech Feature Extractor is trained to identify the synchronization between audio clips and video frames.

A key innovation in this stage is the Memory Module. This module acts as a learned dictionary or buffer that stores multiple potential mapping results between speech and physical movement. By utilizing a memory bank, the SFE can focus on extracting core phonetic and linguistic content, while the memory module handles the variability in how that content might be physically expressed by different speakers or in different contexts.

2. Landmark Prediction via Mixture Density Networks (MDN)

Once the speech features are extracted, the model must predict the facial landmarks that serve as the "skeleton" for the final video. Instead of predicting a single set of coordinates, the authors implement a Mixture Density Network (MDN).

The MDN outputs the parameters of a mixture of Gaussians, representing a probability distribution of possible facial positions. This allows the model to "choose" from a variety of valid natural movements. This approach prevents the stiff, averaged movements common in MSE-based models, leading to more rhythmic and lifelike head tilts and facial expressions.

3. Image-to-Image Translation

The final stage involves rendering the predicted landmarks and the original reference image into photo-realistic video frames. This is achieved using an Image2Image translation network based on the MakeItTalk architecture, ensuring that the structural movements predicted by the MDN are translated into high-quality visual outputs.

Experimental Setup and Results

Datasets

The model was trained and evaluated using the VoxCeleb ecosystem. VoxCeleb2, containing over one million utterances from more than 6,000 celebrities, was used to train the speech feature extractor and the landmark regression model. VoxCeleb1 served as the benchmark for evaluating speaker verification performance.

Quantitative Performance

The framework was compared against leading state-of-the-art (SOTA) methods, including ATVGNet and MakeitTalk. The results demonstrate significant improvements across several key metrics:

  • Speaker Verification: The self-supervised contrastive approach achieved an Equal Error Rate (EER) of 21.04%, outperforming previous contrastive learning baselines (22.09%).
  • Landmark Accuracy: The model achieved a Landmark Distance (LMD) of 1.83, a substantial improvement over MakeitTalk (7.85) and ATVGNet (2.12).
  • Naturalness of Movement: The Rotation Distance (RD), which quantifies the naturalness of head movement, was recorded at 0.07, matching or exceeding the best available SOTA results.

Qualitative Observations

Visual analysis confirms the quantitative findings. While ATVGNet often produces static or "floating" heads and MakeitTalk struggles with precise lip shapes, the proposed model generates pronounced, natural head tilts and highly accurate lip synchronization that aligns with the rhythm of the input audio.

Discussion and Future Work

The core contribution of this work is the dual-level handling of uncertainty: first through the Memory Module during feature extraction, and second through the MDN during landmark generation. This approach acknowledges that multiple head poses are valid for the same syllable, resulting in a more dynamic and less robotic user experience.

Despite these successes, the authors note that the model does not yet perfectly replicate the exact ground truth head movements of a source video, although it captures the correct dynamic trends. Future research will focus on:

  • Emotion Embeddings: Incorporating emotion-specific features to enhance the expressiveness of the generated talking heads.
  • Computational Efficiency: Optimizing the model for real-time applications on edge devices, as the current performance sits at approximately 25 FPS.

Conclusion

By combining memory-augmented contrastive learning with probabilistic landmark prediction, this framework provides a robust solution to the one-to-many mapping problem in talking head generation. The result is a system capable of producing realistic, rhythmic, and synchronized videos from minimal input, setting a new standard for naturalness in the field.

Summary updated

BibTeX
@inproceedings{wang2023memory,
  author = {Jianrong Wang and Yaxin Zhao and Hongkai Fan and Tianyi Xu and Qi Li and Sen Li and Li Liu},
  title = {{Memory-Augmented Contrastive Learning for Talking Head Generation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096593},
}
Figure