Abstract
Talking face synthesis driven by audio is a critical research area in multidimensional signal processing and multimedia. While Neural Radiance Fields (NeRF) have been introduced to enhance 3D realism, existing methods often struggle with complex learning tasks and imprecise mapping of audio to speech-related facial regions, leading to inaccurate lip shapes. This paper proposes NeRF-AD, a method that utilizes an Attention-based Disentanglement module to separate the face into "Audio-face" and "Identity-face" components using facial Action Unit (AU) information. By fusing audio features exclusively with the Audio-face component and supervising the process with AU information, NeRF-AD achieves superior lip synchronization and image quality compared to state-of-the-art methods.
Introduction
Creating realistic talking face videos from audio requires balancing 3D structural consistency with precise temporal lip movements. Standard NeRF-based approaches often suffer from learning overload, where the model attempts to learn static identity and dynamic speech movements simultaneously. Furthermore, inaccurate mapping occurs when audio signals influence static facial regions (like the eyes or forehead), causing unnatural artifacts. NeRF-AD addresses these issues through a "divide and conquer" strategy, disentangling speech-related movements from speaker-specific identity features before the rendering stage.
Methodology
The core of NeRF-AD is a modular pipeline that moves complex feature alignment tasks ahead of the NeRF rendering process. This ensures the radiance field can focus on high-fidelity volumetric rendering while the disentanglement module handles the multimodal mapping.
Attention-based Disentanglement
The model introduces an Attention-based Disentanglement module to split the input face into two streams:
- Audio-face: Regions directly related to speech movements (lips, jaw).
- Identity-face: Static characteristics (eyes, nose, structure) that remain constant regardless of speech.
This is achieved using an AU-Attention module guided by Facial Action Units. As shown in the detailed view below, the system utilizes a Mask Generator and a Warping Generator to isolate speech-related regions unsupervised, creating an attention mask $A$.
Feature Fusion and Supervision
Audio features, extracted via DeepSpeech, are fused only with the Audio-face features. This prevents audio signals from "polluting" identity features, which typically causes jitter in static regions. The fusion is supervised using a specialized AU loss (combining Binary Cross-Entropy and Dice coefficient), ensuring that the generated mouth shapes align with the physical muscle movements expected for specific phonemes.
Conditional NeRF Rendering
The final stage employs a conditional NeRF that takes the fused audio-facial features and the static identity features as inputs. By pre-processing the disentanglement, the NeRF model achieves higher learning efficiency, requiring fewer iterations to produce accurate results.
Experiments and Results
Datasets and Metrics
The researchers evaluated NeRF-AD on high-quality video datasets (25fps, ~7,000 frames per subject). Performance was measured using standard image quality metrics (PSNR, SSIM, LPIPS) and lip-sync metrics (SyncNet, AU Accuracy). Notably, the authors introduced LMD-79, an improved Landmark Distance metric using 79 lip landmarks for more granular accuracy compared to the standard 20.
Performance Analysis
NeRF-AD demonstrated state-of-the-art performance across several benchmarks:
- Lip Synchronization: NeRF-AD achieved an AU Accuracy of 87.5%, outperforming GeneFace (85.2%) and AD-NeRF (86.3%). It also significantly reduced the LMD-79 error to 3.950.
- Image Quality: The model achieved the highest SSIM (0.901) and the lowest LPIPS (0.039) among NeRF-based competitors, indicating sharper textures and fewer artifacts.
- Efficiency: Ablation studies showed that even at low training iterations (4,000 to 10,000), NeRF-AD produced more accurate lip shapes than competing models that had trained for significantly longer periods.
Conclusion
NeRF-AD represents a significant advancement in talking face synthesis by leveraging facial Action Units to guide attention-based disentanglement. By isolating audio influence to specific facial regions, the model avoids common artifacts and achieves superior lip synchronization. Future work may focus on explicitly modeling complex, high-degree-of-freedom head rotations and further reducing the computational overhead inherent in NeRF-based rendering.
Summary updated