SyncTalk++: High-Fidelity and Efficient Synchronized Talking Heads Synthesis Using Gaussian Splatting

Ziqiao Peng, Wentao Hu, Jiao Ma, Xiangyu Zhu, X. M. Zhang, Hao Zhao, Hui Tian, J. He, Hongyan Liu, Zhaoxin FanIEEE TPAMI 2025 · DOI

Abstract

Generating realistic, speech-driven talking head videos requires the seamless synchronization of facial expressions, lip movements, and head poses. Current methods often struggle with the trade-off between rendering speed and 3D consistency, frequently falling into the "uncanny valley." We present SyncTalk++, a novel framework that leverages 3D Gaussian Splatting (3DGS) and a modular synchronization architecture to achieve high-fidelity synthesis at 101 frames per second (FPS). By incorporating a Face-Sync Controller, a Head-Sync Stabilizer, and a VQ-VAE-based expression generator, SyncTalk++ ensures robust performance even with out-of-distribution (OOD) audio, providing a significant leap forward in real-time digital human synthesis.

1. Introduction

The creation of realistic talking heads is a complex balancing act involving four critical factors: subject identity, lip movement, facial expression, and head pose. Discrepancies in any of these dimensions lead to artifacts that the human eye quickly identifies as unnatural. Existing 2D-based methods, such as GANs or Diffusion models, often suffer from 3D inconsistency and facial warping. Conversely, 3D methods based on Neural Radiance Fields (NeRF) offer better structural integrity but are typically too computationally expensive for real-time applications and struggle to synchronize nuanced expressions with speech. SyncTalk++ addresses these challenges by identifying synchronization as the primary hurdle to achieving indistinguishable synthetic video.

2. Methodology

SyncTalk++ introduces a modular architecture designed to tackle synchronization from multiple perspectives, ensuring both efficiency and visual fidelity.

2.1 Dynamic Portrait Renderer (3DGS)

To overcome the latency issues of NeRF, SyncTalk++ utilizes 3D Gaussian Splatting. The system employs a "Triplane Gaussian Representation" to model the head in 3D space. This approach allows for the representation of high-fidelity details while enabling extremely fast rendering speeds, a prerequisite for interactive applications.

2.2 Face-Sync Controller

Unlike generic speech-to-text models, the Face-Sync Controller uses an audio-visual encoder specifically trained to align audio features with lip movements. It incorporates a 3D facial blendshape model, allowing the system to control expressions independently of the speech signal, which enhances the naturalness of the generated performance.

2.3 Head-Sync Stabilizer

To eliminate "head jitter," a common artifact in synthetic video, the Head-Sync Stabilizer employs a two-stage optimization process. A key innovation is the use of Semantic Weighting. By down-weighting unstable or "fidgety" areas (such as blinking eyes or moving eyebrows) during head pose calculations, the system ensures that global head movement remains smooth and natural.

2.4 OOD Audio Expression Generator

Handling Out-of-Distribution (OOD) audio—such as synthesized voices or voices not present in the training set—is a major challenge for talking head models. SyncTalk++ utilizes a VQ-VAE codebook to map unusual audio inputs to a discrete space of realistic facial expressions. This acts as a "sanity check," ensuring that the renderer produces expressions derived from a library of known, realistic 3D blendshapes rather than crashing or producing distorted artifacts.

2.5 Torso Restorer

To ensure a seamless transition between the generated head and the original video background, a lightweight U-Net inpainter is used. This Torso Restorer fixes "seams" or gaps that appear when significant jaw movement occurs, maintaining the integrity of the subject's overall appearance.

3. Experiments and Results

3.1 Datasets

The framework was evaluated using the LRS2 dataset for pre-training the audio-visual synchronization encoder. For individual subject synthesis, standard talking head benchmarks (including English and French sequences) from previous state-of-the-art (SOTA) works like AD-NeRF and GeneFace were utilized.

3.2 Performance and Quality

SyncTalk++ achieves a rendering speed of 101 FPS on an NVIDIA RTX 4090, making it viable for real-time digital assistants and VR environments. Quantitatively, the model outperformed existing SOTA methods, such as TalkingGaussian and ER-NeRF, across nearly all metrics. Notably, it achieved a 65.67% lower error in LPIPS (Learned Perceptual Image Patch Similarity) compared to previous leading models.

3.3 User Study

A study involving 42 participants confirmed the quantitative findings. SyncTalk++ consistently received the highest scores for lip-sync accuracy, expression naturalness, and overall video realism compared to baseline methods.

4. Discussion and Limitations

While SyncTalk++ represents a significant advancement, it is not without limitations. The model currently requires approximately 4-5 minutes of high-quality video of a specific individual to create a high-fidelity "digital twin." Additionally, extremely rapid or wide-angle head rotations can occasionally cause the underlying 3DGS primitives to become visible as "splats."

Furthermore, the authors acknowledge the ethical implications of high-fidelity talking head synthesis. The potential for "deepfake" misuse necessitates the development of robust detection algorithms and digital watermarking techniques to protect individual identities and ensure the responsible use of this technology.

Summary updated

Abstract

Achieving high synchronization in the synthesis of realistic, speech-driven talking head videos presents a significant challenge. A lifelike talking head requires synchronized coordination of subject identity, lip movements, facial expressions, and head poses. The absence of these synchronizations is a fundamental flaw, leading to unrealistic results. To address the critical issue of synchronization, identified as the "devil" in creating realistic talking heads, we introduce SyncTalk++, which features a Dynamic Portrait Renderer with Gaussian Splatting to ensure consistent subject identity preservation and a Face-Sync Controller that aligns lip movements with speech while innovatively using a 3D facial blendshape model to reconstruct accurate facial expressions. To ensure natural head movements, we propose a Head-Sync Stabilizer, which optimizes head poses for greater stability. Additionally, SyncTalk++ enhances robustness to out-of-distribution (OOD) audio by incorporating an Expression Generator and a Torso Restorer, which generate speech-matched facial expressions and seamless torso regions. Our approach maintains consistency and continuity in visual details across frames and significantly improves rendering speed and quality, achieving up to 101 frames per second. Extensive experiments and user studies demonstrate that SyncTalk++ outperforms state-of-the-art methods in synchronization and realism.
BibTeX
@article{peng2025synctalk,
  author = {Ziqiao Peng and Wentao Hu and Jiao Ma and Xiangyu Zhu and X. M. Zhang and Hao Zhao and Hui Tian and J. He and Hongyan Liu and Zhaoxin Fan},
  title = {{SyncTalk++: High-Fidelity and Efficient Synchronized Talking Heads Synthesis Using Gaussian Splatting}},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  doi = {10.1109/tpami.2025.3630057},
}

Loading related content...

Figure