Abstract
Talking head synthesis aims to generate lip-synchronized video from audio input. While Neural Radiance Fields (NeRF) have improved realism and texture details, existing methods often struggle with rendering clear talking heads in novel (non-frontal) views and aligning acoustic signals with visual motion. To address these challenges, we propose NeRF-3DTalker, a framework that utilizes 3D prior information to synthesize high-fidelity talking heads with free-view capabilities. Our approach features a 3D Prior Aided Audio Disentanglement module to separate speech movements from individual speaking styles and a local-global Standardized Space to regularize facial geometry. Experimental results demonstrate that NeRF-3DTalker achieves superior image quality and lip synchronization compared to state-of-the-art methods.
Introduction
Synthesizing realistic talking heads is a complex task requiring precise coordination between 1D audio signals and 3D facial dynamics. Despite the success of NeRF in static scene reconstruction, its application to dynamic talking heads faces two primary hurdles:
- The Frontal View Trap: Most current models are optimized for frontal perspectives. When the virtual camera shifts to a side profile, image quality often degrades, and the underlying 3D structure can collapse.
- Audio-Visual Mismatch: Audio signals are inherently noisy for motion modeling, containing pitch and timbre information that does not directly correlate with lip movement. This misalignment often leads to "mushy" or lagging animations.
NeRF-3DTalker overcomes these limitations by integrating 3D structural priors and a novel disentanglement strategy to ensure consistency across all viewing angles.
Methodology
The NeRF-3DTalker pipeline consists of three core components designed to bridge the gap between audio and 3D geometry:
3D Prior Extraction
The model utilizes a 3D Morphable Model (3DMM) to extract structural priors, including identity ($f_{id}$), albedo ($f_{alb}$), and expression ($f_{illu}$). These priors provide a geometric "skeleton" that helps the NeRF maintain structural integrity and consistency, even when viewed from novel angles.
3D Prior Aided Audio Disentanglement
Rather than processing raw audio directly, we introduce a disentanglement module that splits the signal into two distinct features:
- $f_{exp-aud}$: Features specifically mapped to 3D-awarded speech movements.
- $f_{exp-style}$: Features capturing the speaker's unique stylistic nuances.
This module is supervised by LipNet and SyncNet to ensure the acoustic space is perfectly aligned with the 3D visual space, effectively acting as a denoiser for motion-irrelevant audio data.
Local-Global Standardized Space
To prevent the generation of anatomically impossible facial configurations, we implement a codebook-based regularization system. A Global Encoder manages the overall face structure, while an AU (Action Unit) Encoder focuses on local semantic movements like lip curls. By mapping generated features to a "Standardized Space" of valid facial positions, the model ensures realistic outputs during complex speech sequences.
Experimental Results
Quantitative Evaluation
NeRF-3DTalker was evaluated against several state-of-the-art (SOTA) models, including DFRF, GeneFace, and Synctalk. Our method achieved superior performance in image quality metrics such as SSIM (Structural Similarity) and LPIPS (Perceptual Loss). Notably, on Dataset B, it reached a 78.4% AU Acc (Action Unit Accuracy), significantly outperforming DFRF (75.7%) and GeneFace (75.4%). The improvement in LMD-79 (Landmark Distance) further confirms that the synthesized lip shapes are anatomically accurate.
Qualitative and Multi-View Analysis
A key strength of NeRF-3DTalker is its ability to maintain high-fidelity results in novel views. While competing models often produce blurry or distorted textures at 45-degree angles, our framework preserves crisp lip boundaries and stable facial geometry. This is attributed to the robust alignment between audio features and 3D geometric priors.
Conclusion
NeRF-3DTalker represents a significant advancement in talking head synthesis by successfully integrating 3D priors with audio disentanglement. By regularizing the motion space through a Standardized Space, the model achieves unprecedented clarity in multi-view synthesis. Future work will explore extending this framework to support explicit emotional controls and reducing the computational overhead associated with NeRF rendering to enable real-time applications.
Summary updated