NeRF-3DTalker: Neural Radiance Field with 3D Prior Aided Audio Disentanglement for Talking Head Synthesis

Xiaoxing Liu, Zhilei Liu, Chongke Bi

TL;DR

NeRF-3DTalker is a novel framework that generates high-fidelity, lip-synchronized talking head videos capable of being viewed from any angle. By disentangling audio into distinct "speech movement" and "speaking style" components and utilizing a "Standardized Space" to regularize facial geometry, it overcomes the common "frontal-only" limitation and blurry lip-syncing issues found in previous NeRF-based methods.

What Problem Does This Paper Address?

While Neural Radiance Fields (NeRF) have revolutionized 3D scene reconstruction, applying them to talking heads—where a face must move realistically in sync with audio—remains difficult. Current methods face two primary hurdles:

  1. The Frontal View Trap: Most models are optimized for frontal faces. When the virtual camera moves to a side profile (novel views), the image quality degrades, and the 3D structure often collapses.
  2. Audio-Visual Mismatch: Audio signals are "noisy" for a motion model; they contain rhythm, pitch, and timbre that don't necessarily correlate with lip movement. This makes it hard for a model to learn precise lip-syncing, often resulting in "mushy" or lagging animations.

Methods

The authors introduce a multi-stage pipeline to bridge the gap between 1D audio and 3D facial geometry:

  • 3D Prior Extraction: The model uses a 3D Morphable Model (3DMM) to extract structural priors (identity, albedo, and expression). This provides a geometric "skeleton" that helps the NeRF maintain consistency across different viewing angles.
  • 3D Prior Aided Audio Disentanglement: Instead of feeding raw audio into the NeRF, the researchers propose a module that splits the audio into two features:
  • $f_{exp-aud}$: Features specifically related to 3D-awarded speech movements.
  • $f_{exp-style}$: Features capturing the individual's unique speaking style. This module is supervised by LipNet and SyncNet to ensure the acoustic space aligns perfectly with the 3D visual space.

  • Local-Global Standardized Space: To prevent the model from generating "impossible" facial configurations, they implement a codebook-based regularization system.

  • A Global Encoder handles the overall face structure.
  • An AU (Action Unit) Encoder focuses on local semantic movements (like lip curls or eyebrow raises). By mapping generated features to a "Standardized Space" of known valid facial positions, the model ensures the output remains realistic even during complex speech.

Datasets

The authors collected four high-quality video datasets featuring different individuals giving speeches, interviews, or reading news. These videos were processed to preserve facial head segments and were split into two test groups (TestA and TestB) for evaluation. They followed the data preparation protocols established in previous SOTA works like GeneFace and DFRF.

Key Insights and Ideas

  • Disentanglement as a Denoiser: By separating "style" from "content" in the audio, the model doesn't get "confused" by the speaker's tone or pitch when trying to figure out how the lips should move.
  • Geometry over Pixels: Aligning audio to 3D visual priors (3DMM) is more robust than aligning it directly to 2D pixels. This is why the model maintains such high clarity in profile views.
  • The "Reality Check" Codebook: The Standardized Space acts as a learned constraint. If the NeRF tries to render a frame that is anatomically "weird," the codebook pulls the features back toward a standardized, realistic facial expression.

Key Results

NeRF-3DTalker achieves state-of-the-art (SOTA) performance across several metrics:

  • Image Quality: Outperforms previous models like Synctalk and NeRF-AD in SSIM (Structural Similarity) and LPIPS (Perceptual Loss), indicating sharper textures and fewer artifacts.
  • Lip-Sync Accuracy: On Dataset B, it achieved a 78.4% AU Acc (Action Unit Accuracy), a significant jump over DFRF (75.7%) and GeneFace (75.4%).
  • Geometric Precision: It showed a marked improvement in LMD-79 (Landmark Distance with 79 points), proving that the lip shapes are not just "moving" but are anatomically accurate to the speech.
  • Multi-View Consistency: Qualitative results show that while competitors become blurry or distorted at 45-degree angles, NeRF-3DTalker maintains crisp lip boundaries and facial structures.

Limitations and Future Work

While the paper demonstrates impressive results, a few areas remain for exploration:

  • Personalization: Like many NeRF-based talking head models, this appears to be a "person-specific" model, meaning it likely requires training on a specific individual to achieve these high-fidelity results for that person.
  • Computational Load: Although they mention an upsampling operation ($H_u$) to decrease rendering time, NeRF-based synthesis is generally more computationally expensive than 2D GAN-based methods.
  • Emotional Range: The current model focuses on speech and style; future work could integrate explicit emotional controls (e.g., making the talking head look angry or happy) while maintaining the 3D structural integrity.

Summary updated

Abstract

Talking head synthesis is to synthesize a lip-synchronized talking head video using audio. Recently, the capability of NeRF to enhance the realism and texture details of synthesized talking heads has attracted the attention of researchers. However, most current NeRF methods based on audio are exclusively concerned with the rendering of frontal faces. These methods are unable to generate clear talking heads in novel views. Another prevalent challenge in current 3D talking head synthesis is the difficulty in aligning acoustic and visual spaces, which often results in suboptimal lip-syncing of the generated talking heads. To address these issues, we propose Neural Radiance Field with 3D Prior Aided Audio Disentanglement for Talking Head Synthesis (NeRF-3DTalker). Specifically, the proposed method employs 3D prior information to synthesize clear talking heads with free views. Additionally, we propose a 3D Prior Aided Audio Disentanglement module, which is designed to disentangle the audio into two distinct categories: features related to 3D awarded speech movements and features related to speaking style. Moreover, to reposition the generated frames that are distant from the speaker's motion space in the real space, we have devised a local-global Standardized Space. This method normalizes the irregular positions in the generated frames from both global and local semantic perspectives. Through comprehensive qualitative and quantitative experiments, it has been demonstrated that our NeRF-3DTalker outperforms state-of-the-art in synthesizing realistic talking head videos, exhibiting superior image quality and lip synchronization. Project page: https://nerf-3dtalker.github.io/NeRF-3Dtalker.
BibTeX
@inproceedings{liu2025nerf,
  author = {Xiaoxing Liu and Zhilei Liu and Chongke Bi},
  title = {{NeRF-3DTalker: Neural Radiance Field with 3D Prior Aided Audio Disentanglement for Talking Head Synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10889538},
}

Loading related content...

Figure