Abstract
LokiTalk is a novel framework designed to enhance the realism and efficiency of NeRF-based talking head synthesis. By decomposing facial motion into region-specific deformation fields and implementing an ID-aware knowledge transfer module, LokiTalk addresses common issues such as visual artifacts and high data requirements. Experimental results demonstrate that LokiTalk achieves state-of-the-art visual quality and motion accuracy, while significantly reducing the training data needed for new individuals by up to 75%.
Introduction
Creating realistic talking heads from audio—where a digital avatar speaks and moves naturally—is a significant challenge in digital media. While Neural Radiance Fields (NeRF) have improved 3D consistency, current models face two major hurdles: visual artifacts and training inefficiency. Models often struggle with precise mapping, leading to "mushy" lip-sync or disconnected head-torso movements. Furthermore, standard NeRF models typically require extensive training data for every new identity, making them computationally expensive. LokiTalk addresses these challenges through hierarchical deformation fields and a "plug-and-play" ID-aware knowledge transfer module.
Methodology
LokiTalk introduces two primary innovations: region-specific deformation fields and ID-aware knowledge transfer.
Region-Specific Deformation Fields
Instead of a global deformation field, LokiTalk employs a hierarchical, two-stage approach to model facial and torso movements separately but interdependently.
- Face Deformation: This field is driven by audio signals for lip movements and eye-aspect ratios for blinking.
- Torso Deformation: Crucially, torso movement is driven by both the head pose and the results of the face deformation. This dependency mimics physical reality, where jaw movements influence the neck and upper torso.
- Regularization: Region-specific masks and attention-based loss functions ensure that driving signals only influence their intended areas (e.g., audio only affecting the mouth), preventing "signal leakage" that causes artifacts.
ID-Aware Knowledge Transfer
To improve training efficiency, LokiTalk leverages common human traits through a pre-training phase.
- Pre-training: An ID-Encoder is trained on multi-identity videos to learn universal human motion and geometry.
- Feature Extraction: The module extracts ID-specific static features (e.g., skin color, face shape) and dynamic features (e.g., unique expression styles).
- Fine-tuning: When adapting to a new individual, the model utilizes this rich prior knowledge, allowing for high-fidelity results even with very short target videos.
Experiments and Results
The researchers evaluated LokiTalk using standard datasets with videos averaging 4 minutes in length. Comparisons were made against several baselines, including Wav2Lip, AD-NeRF, RAD-NeRF, and ER-NeRF.
Quantitative Performance
LokiTalk outperforms existing state-of-the-art (SOTA) models across multiple metrics. As shown in the quantitative analysis, it achieved a PSNR of 33.744 and an LPIPS of 0.029, indicating superior visual quality. Additionally, its SyncNet score of 5.736 demonstrates improved lip-sync accuracy compared to other NeRF-based methods.
Data Efficiency and Depth Realism
A key finding is LokiTalk's data efficiency: using only 25% of the training data for a specific person, it still outperformed ER-NeRF trained on 100% of the data. Qualitatively, depth maps reveal that LokiTalk captures finer details in the mouth and eyes compared to the smoother, less detailed maps of earlier models.
Conclusion
LokiTalk significantly advances NeRF-based talking head synthesis by improving both realism and efficiency. While the model achieves high-quality results at 29 FPS, some limitations remain, such as artifacts near hair edges due to segmentation jitter and the requirement for high-end GPUs for optimal performance. Future work may focus on further optimizing the model for real-time applications on consumer-grade hardware and refining segmentation processes to eliminate remaining visual jitters.
Summary updated