Towards Generating Ultra-High Resolution Talking-Face Videos with Lip synchronization

Anchit Gupta, Rudrabha Mukhopadhyay, Sindhu Balachandra, Faizan Farooq Khan, Vinay P. Namboodiri, C. V. JawaharWACV 2023 · DOI

Introduction

Talking-face generation—the task of synchronizing a person's lip movements to an arbitrary audio track—is a critical component in digital media, gaming, and telepresence. However, current state-of-the-art models, such as Wav2Lip, are typically constrained to low resolutions (e.g., 96×96 or 256×256 pixels). Scaling these models to ultra-high resolution (UHR), such as 4K, presents significant challenges in terms of computational requirements, data scarcity, and maintaining temporal coherence. Simply applying super-resolution (SR) to low-resolution outputs often fails to recover fine-grained anatomical details like individual teeth or lip textures.

Methodology

The core innovation of this work is a two-stage pipeline that operates within a compact Vector Quantized (VQ) latent space rather than the standard RGB pixel space.

Stage 1: Latent Space Lip-sync

By utilizing a VQGAN codebook, the model represents a 256×256 image as a 16×16 grid of latent codes. This compression allows the network to focus on learning complex lip dynamics without the overhead of pixel-level redundancy. To ensure accuracy, the researchers adapted a "sync-expert" discriminator to operate entirely within this quantized space.

Stage 2: Pose and Restoration

A Pose-VQGAN encoder provides head pose information by analyzing only the upper half of the face, preventing the model from leaking ground-truth mouth information during training. For the final output, a face restoration network (GPEN) sharpens the results and integrates the generated face back into the high-resolution source frame.

The 4K Talking Faces (4KTF) Dataset

To address the lack of high-resolution training data, the researchers introduced the 4K Talking Faces (4KTF) dataset. This benchmark contains approximately 30 hours of 4K video across 140 diverse identities and 2.5 million frames, covering a wide range of English vocabulary and speaking styles.

Key Results and Insights

The proposed method achieves 64 times the pixel density of previous baselines while maintaining superior synchronization accuracy.

Comparison of the proposed pipeline against the current state-of-the-art, demonstrating superior resolution and detail in the mouth region.

Quantitative Performance

On the 4KTF dataset, the model achieved a Lip Sync Error - Confidence (LSE-C) score of 7.10, nearly doubling the performance of a Wav2Lip-orig + TecoGAN baseline (4.03). Human evaluations also favored the proposed method, particularly in categories of sharpness and overall visual experience.

Qualitative Improvements

Operating in the VQ space allows the model to capture "micro-details," such as the moisture on teeth and subtle lip wrinkles, which are typically lost in upsampling. Furthermore, the model demonstrates robustness in handling silent regions, correctly generating closed-lip shapes where previous models often exhibited "quivering" artifacts.

Discussion and Ethical Considerations

While the framework enables high-fidelity synthesis, the authors acknowledge the risks associated with deepfake technology. To mitigate these risks, they implemented an invisible watermarking technique in the frequency space of generated images, allowing for reliable detection of synthesized content. Future work will focus on reducing the computational load to enable real-time 4K generation on consumer hardware.

Summary updated

BibTeX
@inproceedings{gupta2023towards,
  author = {Anchit Gupta and Rudrabha Mukhopadhyay and Sindhu Balachandra and Faizan Farooq Khan and Vinay P. Namboodiri and C. V. Jawahar},
  title = {{Towards Generating Ultra-High Resolution Talking-Face Videos with Lip synchronization}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00518},
}
Figure