NeRF-AD: Neural Radiance Field with Attention-based Disentanglement for Talking Face Synthesis

Chongke Bi, Xiaoxing Liu, Zhilei Liu

Abstract

Talking face synthesis driven by audio is a critical research area in multidimensional signal processing and multimedia. While Neural Radiance Fields (NeRF) have been introduced to enhance 3D realism, existing methods often struggle with complex learning tasks and imprecise mapping of audio to speech-related facial regions, leading to inaccurate lip shapes. This paper proposes NeRF-AD, a method that utilizes an Attention-based Disentanglement module to separate the face into "Audio-face" and "Identity-face" components using facial Action Unit (AU) information. By fusing audio features exclusively with the Audio-face component and supervising the process with AU information, NeRF-AD achieves superior lip synchronization and image quality compared to state-of-the-art methods.

Introduction

Creating realistic talking face videos from audio requires balancing 3D structural consistency with precise temporal lip movements. Standard NeRF-based approaches often suffer from learning overload, where the model attempts to learn static identity and dynamic speech movements simultaneously. Furthermore, inaccurate mapping occurs when audio signals influence static facial regions (like the eyes or forehead), causing unnatural artifacts. NeRF-AD addresses these issues through a "divide and conquer" strategy, disentangling speech-related movements from speaker-specific identity features before the rendering stage.

Methodology

The core of NeRF-AD is a modular pipeline that moves complex feature alignment tasks ahead of the NeRF rendering process. This ensures the radiance field can focus on high-fidelity volumetric rendering while the disentanglement module handles the multimodal mapping.

The overall architecture of NeRF-AD, illustrating the flow from disentanglement and feature fusion to the final NeRF rendering stage.

Attention-based Disentanglement

The model introduces an Attention-based Disentanglement module to split the input face into two streams:

  • Audio-face: Regions directly related to speech movements (lips, jaw).
  • Identity-face: Static characteristics (eyes, nose, structure) that remain constant regardless of speech.

This is achieved using an AU-Attention module guided by Facial Action Units. As shown in the detailed view below, the system utilizes a Mask Generator and a Warping Generator to isolate speech-related regions unsupervised, creating an attention mask $A$.

Detailed view of the Attention-based Disentanglement module, showing the use of Mask and Warping Generators to isolate speech-related facial regions.

Feature Fusion and Supervision

Audio features, extracted via DeepSpeech, are fused only with the Audio-face features. This prevents audio signals from "polluting" identity features, which typically causes jitter in static regions. The fusion is supervised using a specialized AU loss (combining Binary Cross-Entropy and Dice coefficient), ensuring that the generated mouth shapes align with the physical muscle movements expected for specific phonemes.

Conditional NeRF Rendering

The final stage employs a conditional NeRF that takes the fused audio-facial features and the static identity features as inputs. By pre-processing the disentanglement, the NeRF model achieves higher learning efficiency, requiring fewer iterations to produce accurate results.

Experiments and Results

Datasets and Metrics

The researchers evaluated NeRF-AD on high-quality video datasets (25fps, ~7,000 frames per subject). Performance was measured using standard image quality metrics (PSNR, SSIM, LPIPS) and lip-sync metrics (SyncNet, AU Accuracy). Notably, the authors introduced LMD-79, an improved Landmark Distance metric using 79 lip landmarks for more granular accuracy compared to the standard 20.

Performance Analysis

NeRF-AD demonstrated state-of-the-art performance across several benchmarks:

  • Lip Synchronization: NeRF-AD achieved an AU Accuracy of 87.5%, outperforming GeneFace (85.2%) and AD-NeRF (86.3%). It also significantly reduced the LMD-79 error to 3.950.
  • Image Quality: The model achieved the highest SSIM (0.901) and the lowest LPIPS (0.039) among NeRF-based competitors, indicating sharper textures and fewer artifacts.
  • Efficiency: Ablation studies showed that even at low training iterations (4,000 to 10,000), NeRF-AD produced more accurate lip shapes than competing models that had trained for significantly longer periods.

Conclusion

NeRF-AD represents a significant advancement in talking face synthesis by leveraging facial Action Units to guide attention-based disentanglement. By isolating audio influence to specific facial regions, the model avoids common artifacts and achieves superior lip synchronization. Future work may focus on explicitly modeling complex, high-degree-of-freedom head rotations and further reducing the computational overhead inherent in NeRF-based rendering.

Summary updated

Abstract

Talking face synthesis driven by audio is one of the current research hotspots in the fields of multidimensional signal processing and multimedia. Neural Radiance Field (NeRF) has recently been brought to this research field in order to enhance the realism and 3D effect of the generated faces. However, most existing NeRF-based methods either burden NeRF with complex learning tasks while lacking methods for supervised multimodal feature fusion, or cannot precisely map audio to the facial region related to speech movements. These reasons ultimately result in existing methods generating inaccurate lip shapes. This paper moves a portion of NeRF learning tasks ahead and proposes a talking face synthesis method via NeRF with attention-based disentanglement (NeRF-AD). In particular, an Attention-based Disentanglement module is introduced to disentangle the face into Audio-face and Identity-face using speech-related facial action unit (AU) information. To precisely regulate how audio affects the talking face, we only fuse the Audio-face with audio feature. In addition, AU information is also utilized to supervise the fusion of these two modalities. Extensive qualitative and quantitative experiments demonstrate that our NeRF-AD outperforms state-of-the-art methods in generating realistic talking face videos, including image quality and lip synchronization. To view video results, please refer to https://xiaoxingliu02.github.io/NeRF-AD.
BibTeX
@inproceedings{bi2024nerf,
  author = {Chongke Bi and Xiaoxing Liu and Zhilei Liu},
  title = {{NeRF-AD: Neural Radiance Field with Attention-based Disentanglement for Talking Face Synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=dKOMCwFP7l},
}
Figure