Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition

Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, Jingdong Wang

TL;DR

Researchers have developed RAD-NeRF, a framework that achieves real-time (40+ FPS) talking portrait synthesis—a staggering 500x speedup over previous state-of-the-art NeRF methods. The secret lies in decomposing high-dimensional audio-driven facial dynamics into separate, low-dimensional feature grids, allowing for efficient training and lightning-fast inference without sacrificing photorealistic quality.

What Problem Does This Paper Address?

Neural Radiance Fields (NeRF) have revolutionized 3D scene reconstruction, but applying them to "talking heads" (audio-driven portraits) has been notoriously slow. While models like AD-NeRF produce high-quality results, they often take days to train and several seconds to render a single frame, making them useless for real-time applications like virtual assistants or teleconferencing.

The primary bottleneck is the "curse of dimensionality." To make a NeRF "talk," you must condition the 3D spatial coordinates on high-dimensional audio signals. Recent "grid-based" NeRFs (like Instant-NGP) speed up static scenes by storing features in a grid, but adding audio dimensions to these grids causes the computational cost of interpolation to explode exponentially.

Methods

The authors introduce Real-time Audio-spatial Decomposed NeRF (RAD-NeRF). The architecture is built on three key innovations:

  1. Decomposed Audio-spatial Encoding: Instead of querying one high-dimensional grid, the model splits the head representation into two: a 3D spatial grid ($E^3_{spatial}$) and a 2D audio grid ($E^2_{audio}$). By compressing audio features into a 2D "audio coordinate" that depends on the spatial location, they reduce interpolation costs from $2^{3+D}$ to $2^3 + 2^D$.
  2. Pseudo-3D Deformable Torso: Modeling the torso with a full 3D NeRF is overkill. RAD-NeRF uses a lightweight 2D feature grid with a deformation field conditioned on head pose. This ensures the body moves naturally with the head while remaining computationally cheap.
  3. Explicit Control Modules: The system includes an explicit eye-blink control based on an "eye ratio" derived from 2D landmarks, preventing the "half-blink" artifacts common in purely audio-driven models.
The RAD-NeRF architecture decomposes the head into spatial and audio grids while using a pseudo-3D module for the torso.

Datasets

The model was evaluated on standard datasets for talking head synthesis:

  • Obama Dataset: A 3-5 minute video of Barack Obama used for self-driven evaluation.
  • NVP & SynObama: Used for cross-driven evaluation (using one person's audio to drive another's face).
  • Preprocessing: All videos underwent semantic parsing (to separate head/torso), 2D landmark extraction, and face tracking to estimate head pose.

Key Insights and Ideas

  • Efficiency through Decomposition: The most significant "aha moment" is that audio-spatial information doesn't need to be entangled in a high-dimensional space. Separating them into lower-dimensional grids makes grid-based acceleration viable for dynamic scenes.
  • Spatial-Dependent Audio Coordinates: By making the audio coordinate $x_a$ a function of both the audio signal and the spatial feature $f$, the model can represent complex mouth movements more compactly.
  • Maximum Occupancy Grid Pruning: Standard NeRFs use occupancy grids to skip empty space. The authors found that for talking heads, the occupancy doesn't change much with audio. They maintain a "maximum occupancy grid" across all audio conditions, allowing for aggressive ray-marching optimization.

Key Results

RAD-NeRF outperforms previous methods across the board:

  • Speed: It achieves 40 FPS on an NVIDIA V100 GPU, compared to AD-NeRF’s 0.08 FPS (a 500x improvement).
  • Training Time: Training takes roughly 7 hours total (5 for head, 2 for torso), whereas previous methods required 36+ hours.
  • Quality: Quantitatively, it scores higher in PSNR (image quality) and lower in Landmark Distance (LMD), indicating more accurate lip-syncing.
Figure 3. Cross-driven quality comparison. We show visualizations of representative methods under the cross-driven setting. Yellow boxes denote low image quality, and red boxes denote inaccurate lips. Our methods generate both good image quality and accurate lips movement. We recommend watching the supplementary video for better details.

The model also allows for explicit manipulation of head pose, eye blinks, and background images during inference.

Figure 5. Explicit control of talking portrait synthesis. Apart from lips, our method also supports explicit control of eyes, head poses, and background images.

Limitations and Future Work

Despite its speed, RAD-NeRF has a few hurdles:

  • Semantic Parsing Dependency: The quality relies heavily on accurate segmentation of the head and torso.
  • Hair Artifacts: The model currently struggles with long, shoulder-length hair that moves significantly, as it can get "tangled" between the head and torso modules.
  • Language Sensitivity: Because it uses an English-based ASR (Automatic Speech Recognition) model, lip accuracy may drop when driven by non-English audio. The authors suggest moving toward audio-to-phone models to solve this.

Summary updated

Abstract

While dynamic Neural Radiance Fields (NeRF) have shown success in high-fidelity 3D modeling of talking portraits, the slow training and inference speed severely obstruct their potential usage. In this paper, we propose an efficient NeRF-based framework that enables real-time synthesizing of talking portraits and faster convergence by leveraging the recent success of grid-based NeRF. Our key insight is to decompose the inherently high-dimensional talking portrait representation into three low-dimensional feature grids. Specifically, a Decomposed Audio-spatial Encoding Module models the dynamic head with a 3D spatial grid and a 2D audio grid. The torso is handled with another 2D grid in a lightweight Pseudo-3D Deformable Module. Both modules focus on efficiency under the premise of good rendering quality. Extensive experiments demonstrate that our method can generate realistic and audio-lips synchronized talking portrait videos, while also being highly efficient compared to previous methods.
BibTeX
@article{tang2025real,
  author = {Jiaxiang Tang and Kaisiyuan Wang and Hang Zhou and Xiaokang Chen and Dongliang He and Tianshu Hu and Jingtuo Liu and Gang Zeng and Jingdong Wang},
  title = {{Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition}},
  journal = {Int. J. Comput. Vis.},
  year = {2025},
  doi = {10.1007/s11263-025-02481-9},
}
Figure