NeRF-3DTalker: Neural Radiance Field with 3D Prior Aided Audio Disentanglement for Talking Head Synthesis

Xiaoxing Liu, Zhilei Liu, Chongke Bi

Abstract

Talking head synthesis aims to generate lip-synchronized video from audio input. While Neural Radiance Fields (NeRF) have improved realism and texture details, existing methods often struggle with rendering clear talking heads in novel (non-frontal) views and aligning acoustic signals with visual motion. To address these challenges, we propose NeRF-3DTalker, a framework that utilizes 3D prior information to synthesize high-fidelity talking heads with free-view capabilities. Our approach features a 3D Prior Aided Audio Disentanglement module to separate speech movements from individual speaking styles and a local-global Standardized Space to regularize facial geometry. Experimental results demonstrate that NeRF-3DTalker achieves superior image quality and lip synchronization compared to state-of-the-art methods.

Introduction

Synthesizing realistic talking heads is a complex task requiring precise coordination between 1D audio signals and 3D facial dynamics. Despite the success of NeRF in static scene reconstruction, its application to dynamic talking heads faces two primary hurdles:

  1. The Frontal View Trap: Most current models are optimized for frontal perspectives. When the virtual camera shifts to a side profile, image quality often degrades, and the underlying 3D structure can collapse.
  2. Audio-Visual Mismatch: Audio signals are inherently noisy for motion modeling, containing pitch and timbre information that does not directly correlate with lip movement. This misalignment often leads to "mushy" or lagging animations.

NeRF-3DTalker overcomes these limitations by integrating 3D structural priors and a novel disentanglement strategy to ensure consistency across all viewing angles.

Methodology

Overview of the NeRF-3DTalker architecture, including 3D Prior Extraction, Audio Disentanglement, and the Standardized Space module.

The NeRF-3DTalker pipeline consists of three core components designed to bridge the gap between audio and 3D geometry:

3D Prior Extraction

The model utilizes a 3D Morphable Model (3DMM) to extract structural priors, including identity ($f_{id}$), albedo ($f_{alb}$), and expression ($f_{illu}$). These priors provide a geometric "skeleton" that helps the NeRF maintain structural integrity and consistency, even when viewed from novel angles.

3D Prior Aided Audio Disentanglement

Rather than processing raw audio directly, we introduce a disentanglement module that splits the signal into two distinct features:

  • $f_{exp-aud}$: Features specifically mapped to 3D-awarded speech movements.
  • $f_{exp-style}$: Features capturing the speaker's unique stylistic nuances.

This module is supervised by LipNet and SyncNet to ensure the acoustic space is perfectly aligned with the 3D visual space, effectively acting as a denoiser for motion-irrelevant audio data.

Local-Global Standardized Space

To prevent the generation of anatomically impossible facial configurations, we implement a codebook-based regularization system. A Global Encoder manages the overall face structure, while an AU (Action Unit) Encoder focuses on local semantic movements like lip curls. By mapping generated features to a "Standardized Space" of valid facial positions, the model ensures realistic outputs during complex speech sequences.

Experimental Results

Quantitative Evaluation

NeRF-3DTalker was evaluated against several state-of-the-art (SOTA) models, including DFRF, GeneFace, and Synctalk. Our method achieved superior performance in image quality metrics such as SSIM (Structural Similarity) and LPIPS (Perceptual Loss). Notably, on Dataset B, it reached a 78.4% AU Acc (Action Unit Accuracy), significantly outperforming DFRF (75.7%) and GeneFace (75.4%). The improvement in LMD-79 (Landmark Distance) further confirms that the synthesized lip shapes are anatomically accurate.

Qualitative and Multi-View Analysis

Qualitative comparison of novel view synthesis between NeRF-AD, Synctalk, and NeRF-3DTalker. Our method maintains clarity and structural integrity at extreme angles.

A key strength of NeRF-3DTalker is its ability to maintain high-fidelity results in novel views. While competing models often produce blurry or distorted textures at 45-degree angles, our framework preserves crisp lip boundaries and stable facial geometry. This is attributed to the robust alignment between audio features and 3D geometric priors.

Conclusion

NeRF-3DTalker represents a significant advancement in talking head synthesis by successfully integrating 3D priors with audio disentanglement. By regularizing the motion space through a Standardized Space, the model achieves unprecedented clarity in multi-view synthesis. Future work will explore extending this framework to support explicit emotional controls and reducing the computational overhead associated with NeRF rendering to enable real-time applications.

Summary updated

Abstract

Talking head synthesis is to synthesize a lip-synchronized talking head video using audio. Recently, the capability of NeRF to enhance the realism and texture details of synthesized talking heads has attracted the attention of researchers. However, most current NeRF methods based on audio are exclusively concerned with the rendering of frontal faces. These methods are unable to generate clear talking heads in novel views. Another prevalent challenge in current 3D talking head synthesis is the difficulty in aligning acoustic and visual spaces, which often results in suboptimal lip-syncing of the generated talking heads. To address these issues, we propose Neural Radiance Field with 3D Prior Aided Audio Disentanglement for Talking Head Synthesis (NeRF-3DTalker). Specifically, the proposed method employs 3D prior information to synthesize clear talking heads with free views. Additionally, we propose a 3D Prior Aided Audio Disentanglement module, which is designed to disentangle the audio into two distinct categories: features related to 3D awarded speech movements and features related to speaking style. Moreover, to reposition the generated frames that are distant from the speaker's motion space in the real space, we have devised a local-global Standardized Space. This method normalizes the irregular positions in the generated frames from both global and local semantic perspectives. Through comprehensive qualitative and quantitative experiments, it has been demonstrated that our NeRF-3DTalker outperforms state-of-the-art in synthesizing realistic talking head videos, exhibiting superior image quality and lip synchronization. Project page: https://nerf-3dtalker.github.io/NeRF-3Dtalker.
BibTeX
@inproceedings{liu2025nerf,
  author = {Xiaoxing Liu and Zhilei Liu and Chongke Bi},
  title = {{NeRF-3DTalker: Neural Radiance Field with 3D Prior Aided Audio Disentanglement for Talking Head Synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10889538},
}
Figure