NERF-AD: Neural Radiance Field With Attention-Based Disentanglement For Talking Face Synthesis

Chongke Bi, Xiaoxing Liu, Zhilei Liu

TL;DR

NeRF-AD introduces a "divide and conquer" strategy for talking face synthesis by disentangling facial features into speech-related "Audio-face" and speaker-specific "Identity-face" components. By using facial Action Units (AUs) to guide an attention mechanism, the model focuses audio influence only where it belongs, resulting in superior lip synchronization and higher image quality with lower training requirements than previous NeRF-based methods.

What Problem Does This Paper Address?

Creating a realistic video of a person talking based only on an audio track is a significant challenge in computer vision. While Neural Radiance Fields (NeRF) have improved the 3D realism of these "talking heads," they often suffer from two main issues:

  1. Learning Overload: Standard NeRF models try to learn the person's entire identity and their complex speech movements simultaneously, which is a heavy computational burden.
  2. Inaccurate Mapping: Many models struggle to precisely map audio signals to the specific regions of the face that actually move during speech (like the lips and jaw). This often results in "mushy" lip movements or artifacts where the audio incorrectly influences static parts of the face.

Methods

The authors propose NeRF-AD (Neural Radiance Field with Attention-based Disentanglement). The core innovation is the Attention-based Disentanglement module, which splits the input face into two distinct streams:

  • Audio-face: This represents the facial regions directly related to speech movements. The model uses an AU-Attention module—guided by Facial Action Units (AUs)—to generate a mask that isolates these areas.
  • Identity-face: This captures the static characteristics of the speaker (eyes, nose, overall structure) that shouldn't change based on what is being said.

The Workflow: 1. Disentanglement: The system uses an unsupervised approach with a Mask Generator and a Warping Generator to create an attention mask $A$. 2. Feature Fusion: Audio features (extracted via DeepSpeech) are fused only with the Audio-face features. This ensures that the audio signal doesn't "pollute" the identity features. 3. Supervision: The fusion process is supervised using a specialized AU loss (Binary Cross-Entropy and Dice coefficient), ensuring the generated mouth shapes match the expected muscle movements for that specific audio. 4. Conditional NeRF: Finally, a conditional NeRF takes both the fused audio-facial features and the identity features as inputs to render the final, high-fidelity video frame.

Datasets

The researchers utilized publicly available video datasets commonly used in talking head research.

  • Data Composition: Seven high-quality videos of different individuals, each at 25fps with approximately 7,000 frames.
  • Split: Five videos were used for general training, while the remaining two were used for fine-tuning and testing.
  • Evaluation Metrics: They used standard image quality metrics (PSNR, SSIM, LPIPS) and lip-sync metrics (SyncNet, AU Accuracy). Notably, they improved the Landmark Distance metric (LMD-79) by increasing the number of lip landmarks from 20 to 79 using MediaPipe for more granular accuracy.

Key Insights and Ideas

  • AUs as a Bridge: Facial Action Units (AUs) are a powerful intermediary. By using them to guide attention, the model learns a "semantic" understanding of which muscles move during speech, rather than just trying to map raw audio to raw pixels.
  • Reducing NeRF's Burden: By moving the disentanglement and fusion tasks ahead of the NeRF rendering stage, the NeRF itself only has to focus on high-quality volumetric rendering rather than complex multimodal feature alignment.
  • Precision Control: Because the audio only affects the "Audio-face" component, the model naturally avoids common artifacts where the speaker's eyes or forehead might jitter in sync with the audio.

Key Results

NeRF-AD demonstrated state-of-the-art performance across several categories:

  • Lip Synchronization: On "Test A," NeRF-AD achieved an AU Accuracy of 87.5%, outperforming GeneFace (85.2%) and AD-NeRF (86.3%). It also significantly reduced the LMD-79 error to 3.950.
  • Image Quality: It achieved the highest SSIM (0.901) and the lowest LPIPS (0.039) among NeRF-based competitors, indicating sharper, more realistic textures.
  • Efficiency: In ablation studies, the model showed high "learning efficiency." Even at only 4,000 or 10,000 training iterations, NeRF-AD produced more accurate lip shapes than other models that had trained longer.

Limitations and Future Work

While the paper demonstrates significant progress, some areas remain for exploration:

  • Head Pose: While the model handles facial movements well, the paper notes that some non-NeRF methods (like Wav2Lip) still struggle with natural head poses; NeRF-AD improves this but doesn't explicitly focus on complex, high-degree-of-freedom head rotations.
  • Computational Cost: Although more efficient than previous NeRF methods, NeRF-based rendering is still generally slower than 2D-only GAN-based approaches.
  • Generalization: The model requires a small amount of fine-tuning for new identities, which is a common hurdle for high-fidelity personalized talking heads.

Summary updated

Abstract

Talking face synthesis driven by audio is one of the current research hotspots in the fields of multidimensional signal processing and multimedia. Neural Radiance Field (NeRF) has recently been brought to this research field in order to enhance the realism and 3D effect of the generated faces. However, most existing NeRF-based methods either burden NeRF with complex learning tasks while lacking methods for supervised multimodal feature fusion, or cannot precisely map audio to the facial region related to speech movements. These reasons ultimately result in existing methods generating inaccurate lip shapes. This paper moves a portion of NeRF learning tasks ahead and proposes a talking face synthesis method via NeRF with attention-based disentanglement (NeRF-AD). In particular, an Attention-based Disentanglement module is introduced to disentangle the face into Audio-face and Identity-face using speech-related facial action unit (AU) information. To precisely regulate how audio affects the talking face, we only fuse the Audio-face with audio feature. In addition, AU information is also utilized to supervise the fusion of these two modalities. Extensive qualitative and quantitative experiments demonstrate that our NeRF-AD outperforms state-of-the-art methods in generating realistic talking face videos, including image quality and lip synchronization. To view video results, please refer to https://xiaoxingliu02.github.io/NeRF-AD/.
BibTeX
@inproceedings{bi2024nerf,
  author = {Chongke Bi and Xiaoxing Liu and Zhilei Liu},
  title = {{NERF-AD: Neural Radiance Field With Attention-Based Disentanglement For Talking Face Synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=dKOMCwFP7l},
}

Loading related content...

Figure