Real-Time Neural Radiance Talking Portrait Synthesis via Audio-Spatial Decomposition

Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Ziwei Liu, Gang Zeng, Jingdong Wang

TL;DR

This paper introduces RAD-NeRF, a novel framework for generating realistic, audio-synchronized talking portraits in real-time. By decomposing the complex audio-spatial representation into efficient low-dimensional grids and employing a lightweight torso modeling approach, RAD-NeRF achieves a significant speedup (up to 500x faster inference) compared to previous NeRF-based methods while maintaining high visual quality.

What Problem Does This Paper Address?

Creating realistic talking portraits from audio has been a long-standing challenge with applications in virtual avatars, digital humans, and more. While recent advancements in Neural Radiance Fields (NeRF) have enabled impressive 3D rendering of dynamic scenes, their application to talking portraits has been hampered by extremely slow training and inference times. Existing methods often require seconds to render a single frame, making them impractical for real-time applications. Furthermore, efficiently modeling both the intricate head movements driven by audio and the more static torso, while maintaining synchronization, presents a significant technical hurdle.

Methods

The core innovation of RAD-NeRF lies in its Audio-Spatial Decomposition approach for modeling the head. Instead of directly encoding high-dimensional audio-spatial features into a single grid, which leads to prohibitive computational complexity, RAD-NeRF decomposes this information into two separate, low-dimensional feature grids:

  • 3D Spatial Grid: Captures the static spatial information of the head.
  • 2D Audio Grid: Encodes the dynamic audio information. Crucially, this audio grid is made spatial-dependent, meaning the audio features are conditioned on the spatial location, helping to prevent undesirable flickering and improve temporal coherence.

For the torso, which is less dynamic, the paper proposes a Pseudo-3D Deformable Module. This module uses a lightweight 2D feature grid and a deformation field conditioned on head pose to synthesize natural torso movements that are synchronized with the head.

To further enhance efficiency, RAD-NeRF leverages grid-based NeRF techniques that replace expensive MLP computations with faster linear interpolations on explicit feature grids. They also introduce a Maximum Occupancy Grid Pruning strategy, which maintains a single 3D occupancy grid for all audio conditions to efficiently prune unnecessary ray sampling.

Additional techniques include:

  • Explicit Eye Control: Using facial landmarks to derive a 1D eye feature that directly conditions the NeRF for better control over eye blinks.
  • Lips Fine-tuning: Employing LPIPS loss specifically on the lip region to improve the accuracy of mouth movements.

Datasets

The paper evaluates its method on datasets collected by previous works, specifically mentioning the Obama dataset from AD-NeRF [22] for self-driven evaluation. For cross-driven evaluation, they use audio clips from NVP [53] and SynObama [50]. No new datasets were introduced.

Key Insights and Ideas

  • Decomposition is Key for Efficiency: The core insight is that directly handling high-dimensional audio-spatial information in grid-based NeRF is computationally infeasible. Decomposing it into separate, lower-dimensional spatial and audio grids is crucial for achieving efficiency.
  • Spatial-Dependent Audio Encoding: Making the audio features spatially dependent is a clever way to ensure that audio cues only influence the relevant parts of the face, preventing artifacts and improving temporal consistency.
  • Lightweight Torso Modeling: Recognizing that the torso is less dynamic than the head allows for a much simpler, pseudo-3D approach, significantly reducing computational load without sacrificing visual coherence.
  • Grid-Based NeRF for Speed: The paper effectively adapts recent grid-based NeRF advancements, which are typically for static scenes, to the dynamic domain of talking portraits.
  • Lips Matter: Explicitly fine-tuning the lip region with perceptual losses highlights the importance of accurate mouth movements for perceived realism.

Key Results

  • Real-time Inference: RAD-NeRF achieves real-time inference speeds, with results showing up to 500x faster than AD-NeRF [22].
  • Faster Training: The method also converges significantly faster, approximately 5x faster than AD-NeRF.
  • High Rendering Quality: The synthesized videos are described as realistic and audio-lips synchronized, with sharper and more accurate lips compared to previous methods.
  • Quantitative Improvements: In self-driven settings, RAD-NeRF achieves better PSNR, LPIPS, and LMD metrics. In cross-driven settings, it demonstrates comparable performance in audio-lips synchronization (Sync↑) and muscle action consistency (AUE↓).
  • Explicit Control: The method allows for explicit control over eye blinks and head poses.

Limitations and Future Work

  • Semantic Parsing Dependency: Like many NeRF-based talking portrait methods, RAD-NeRF relies on accurate semantic parsing to separate the head and torso. Errors in this step can impact results.
  • Hair and Background: The paper notes that synthesizing natural videos for individuals with shoulder-length hair can be challenging, though this is less of an issue in static head pose settings.
  • Language Dependency: The current ASR model is English-specific, which may lead to less accurate lip synchronization when guided by other languages. The authors suggest using audio-to-phone models to address this.
  • Potential Misuse: The authors acknowledge the potential for misuse of realistic talking portrait synthesis technology and express commitment to supporting deepfake detection communities.

Future work could involve extending the method to handle more complex hair dynamics, supporting a wider range of languages, and further improving robustness to challenging visual conditions.

Summary updated

Abstract

Dynamic Neural Radiance Fields (NeRF) have been successful in high-fidelity 3D modeling of talking portraits. However, slow training and inference speed have obstructed their potential usage. This paper proposes an efficient NeRF-based framework, which enables faster convergence and real-time synthesizing of stable talking portraits, by utilizing the recent success of grid-based NeRF. This is accomplished by decomposing the inherently high-dimensional talking portrait representation into three low-dimensional feature grids. Specifically, a Decomposed Audio-Spatial Encoding Module models the dynamic head with a 3D spatial grid and a 2D audio grid, where audio dynamics are modeled in a spatial-dependent manner to avoid undesirable flickering. The torso is handled with another 2D grid in a lightweight Pseudo-3D Deformable Module. Extensive experiments demonstrate that our method can generate realistic and audio-lips synchronized talking portrait videos, while also being highly efficient. Our project page is available at https://me.kiui.moe/radnerf/.
BibTeX
@article{tang2025real,
  author = {Jiaxiang Tang and Kaisiyuan Wang and Hang Zhou and Xiaokang Chen and Dongliang He and Tianshu Hu and Jingtuo Liu and Ziwei Liu and Gang Zeng and Jingdong Wang},
  title = {{Real-Time Neural Radiance Talking Portrait Synthesis via Audio-Spatial Decomposition}},
  journal = {Int. J. Comput. Vis.},
  year = {2025},
  doi = {10.1007/s11263-025-02481-9},
}
Figure