LaDTalk: Latent Denoising for Synthesizing Talking Head Videos with High Frequency Details

Jian Yang, Xukun Wang, Wentao Wang, Guoming Li, Qihang Fang, Ruihong Yuan, Tianyang Wang, Zhaoxin Fan

Abstract

Synthesizing high-fidelity talking head videos requires a delicate balance between precise lip-syncing and rich, realistic textures. Current neural rendering methods often suffer from spectral bias, resulting in blurry outputs that lack high-frequency details like skin pores and hair. While post-processing restoration can mitigate this, it frequently introduces temporal flickering. This paper presents LaDTalk, a novel two-stage pipeline that utilizes a Space-Optimized Vector Quantized AutoEncoder (SOVQAE). By applying Lipschitz Continuity Theory, we demonstrate that VQ-AEs possess an inherent noise robustness that can be optimized for lossless texture restoration. Our system achieves state-of-the-art visual quality and temporal consistency, running at real-time speeds on consumer hardware.

Introduction

The generation of realistic talking heads is a cornerstone of digital human synthesis. However, existing models often struggle with "spectral bias," where neural networks prioritize low-frequency structural information over high-frequency textural details. This leads to the "blurry face" problem common in many AI-generated videos. Traditional solutions involve blind face restoration, but these are typically frame-independent, leading to distracting temporal artifacts. LaDTalk addresses these challenges by reframing texture restoration as a latent denoising problem within a robust discrete space.

Methodology

The Two-Stage Pipeline

LaDTalk employs a cascaded architecture to decouple motion synthesis from texture refinement:

  1. Lip-Sync Prior: The system uses a pretrained Wav2Lip model to generate initial frames. While Wav2Lip provides highly accurate mouth movements, its outputs are often low-resolution and lack fine detail.
  2. Texture Repair (SOVQAE): The core of the system is the Space-Optimized Vector Quantized AutoEncoder. This stage takes the blurry output from the first stage and restores high-frequency details.
The LaDTalk pipeline combining vanilla Wav2Lip with an optimized VQAE for high-frequency detail restoration.

Lipschitz-Driven Noise Robustness

The researchers utilized Lipschitz Continuity Theory to prove that VQ-AEs have an inherent "Noise Robustness Upper Bound" (NRoUB). In a VQ-AE, continuous latent vectors are mapped to the nearest discrete entry in a codebook. If the "noise" (the blurriness or perturbation from the first stage) remains within a specific bound, the mapping remains stable.

Illustration of Voronoi regions and anchor points demonstrating the Noise Robustness Upper Bound (NRoUB) in VQ-AEs.

To maximize this effect, the authors introduced a space regularization loss. This loss pushes codebook entries further apart in the latent space, expanding the Voronoi regions and making the "snapping" mechanism more resilient to input noise. This ensures that the restored textures remain temporally consistent across frames.

Datasets and Training

The model was evaluated using standard dialogue benchmarks (e.g., GeneFace) and a new HF (High-Frequency) Video dataset. This proprietary dataset consists of 4K YouTube videos reprocessed to 1024x1024 resolution, featuring significantly richer textures than standard benchmarks. The SOVQAE is trained identity-specifically, allowing it to "memorize" and perfectly reconstruct the unique high-frequency features of a specific individual.

Key Results

LaDTalk demonstrates significant improvements across several metrics:

  • Visual Fidelity: Achieved state-of-the-art scores in PSNR (38.92) and FID (3.55), outperforming models like SyncTalk and GeneFace++.
  • Temporal Stability: The space-optimized latent mapping effectively eliminates the flickering common in other restoration methods.
  • Efficiency: The pipeline runs at 30 FPS on an NVIDIA 4090 GPU and requires only 5 GPU hours to adapt to a new identity, making it highly practical for real-time applications.

Conclusion

LaDTalk provides a theoretically grounded approach to high-fidelity talking head synthesis. By leveraging the discrete nature of VQ-AEs and optimizing their latent space for noise robustness, the model achieves superior texture restoration without sacrificing temporal consistency. Future work will explore integrating 3D representations, such as Gaussian Splatting, to enable head rotation while maintaining high-frequency detail.

Summary updated

Abstract

Audio-driven IDentity-specific Talking Head Generation (ID-specific THG) has shown increasing promise for applications in filmmaking and virtual reality. Existing approaches are generally constructed as end-to-end paradigms, and have achieved significant progress. However, they often struggle to capture high-frequency textures due to limited model capacity. To address these limitations, we adopt a simple yet efficient post-processing framework -- unlike previous studies that focus solely on end-to-end training -- guided by our theoretical insights. Specifically, leveraging the \textit{Lipschitz Continuity Theory} of neural networks, we prove a crucial noise tolerance property for the Vector Quantized AutoEncoder (VQ-AE), and establish the existence of a Noise Robustness Upper Bound (NRoUB). This insight reveals that we can efficiently obtain an identity-specific denoiser by training an identity-specific neural discrete representation, without requiring an extra network. Based on this theoretical foundation, we propose a plug-and-play Space-Optimized VQ-AE (SOVQAE) with enhanced NRoUB to achieve temporally-consistent denoising. For practical deployment, we further introduce a cascade pipeline combining a pretrained Wav2Lip model with SOVQAE to perform ID-specific THG. Our experiments demonstrate that this pipeline achieves \textit{state-of-the-art} performance in video quality and robustness for out-of-distribution lip synchronization, surpassing existing identity-specific THG methods. In addition, the pipeline requires only a couple of consumer GPU hours and runs in real time, which is both efficient and practical for industry applications.
BibTeX
@misc{yang2024ladtalk,
  author = {Jian Yang and Xukun Wang and Wentao Wang and Guoming Li and Qihang Fang and Ruihong Yuan and Tianyang Wang and Zhaoxin Fan},
  title = {{LaDTalk: Latent Denoising for Synthesizing Talking Head Videos with High Frequency Details}},
  year = {2024},
  eprint = {2410.00990},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.00990},
}
Figure