Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis

Zhenhui Ye, Tianyun Zhong, Yi Ren, Jiaqi Yang, Weichuang Li, Jiawei Huang, Ziyue Jiang, Jinzheng He, Rongjie Huang, Jinglin Liu, Chen Zhang, Xiang Yin, Zejun Ma, Zhou Zhao

Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis

Introduction

Generating realistic talking head videos from a single source image—the "one-shot" setting—is a significant challenge in computer vision. While 2D-based methods can produce high-quality frames, they often fail to maintain 3D consistency, leading to warping artifacts during head rotations. Conversely, existing 3D methods like Neural Radiance Fields (NeRF) typically require extensive per-subject training. Real3D-Portrait addresses these limitations by introducing a framework that lifts a single 2D image into a 3D representation capable of stable, high-fidelity animation driven by either video or audio.

Methodology

The Real3D-Portrait framework is a multi-stage pipeline designed for robust 3D reconstruction and realistic motion synthesis. It decomposes the task into specialized modules for geometry, motion, and rendering.

Overall inference pipeline of Real3D-Portrait, showing the flow from source image and driving resources to the final predicted frames.

1. Image-to-Plane (I2P) Model

The core of the system is the I2P model, which converts a 2D source image into a 3D tri-plane representation. It employs a dual-branch architecture: a Vision Transformer (ViT) branch to capture global 3D structure and a VGG branch for fine-grained texture details. To ensure strong geometric priors, the model is pre-trained on synthetic data from a 3D GAN (EG3D), allowing it to generalize to unseen real-world faces.

2. Motion Adapter (MA)

To animate the 3D head, the Motion Adapter predicts a "residual motion diff-plane." Instead of complex 3D deformations, it uses a Projected Normalized Coordinate Code (PNCC) to represent facial expressions. The MA calculates the necessary updates to the canonical tri-plane, ensuring stable animation and superior identity preservation even during rapid movements.

Detailed architectures of the sub-modules: (a) Image-to-Plane model, (b) Motion Adapter, (c) SegFormer block, and (d) HTB-SR model.

3. Head-Torso-Background (HTB-SR) Model

To create a complete portrait, the system separately models the head, torso, and background. The head is rendered via a 3D volume renderer, while the torso is handled through 2D warping. A KNN-based inpainting module manages the background. These components are fused using alpha-blending and refined by a super-resolution (SR) model to produce a seamless, high-resolution output.

4. Audio-to-Motion (A2M)

For audio-driven synthesis, a VAE-based model converts HuBERT-processed audio into 3DMM expression parameters. These parameters are then mapped to the PNCC format required by the Motion Adapter.

Experiments and Results

The model was trained on the CelebV-HQ and VoxCeleb2 datasets. Quantitative evaluations showed that Real3D-Portrait outperforms state-of-the-art 2D (e.g., Face-vid2vid) and 3D (e.g., HiDe-NeRF) baselines in terms of image quality (FID), identity similarity (CSIM), and lip-sync accuracy (Sync-score). Qualitative results demonstrate the model's ability to maintain structural integrity at large rotation angles, a common failure point for previous methods. A user study further confirmed its superiority in visual quality and synchronization.

Conclusion

Real3D-Portrait presents a robust solution for one-shot 3D talking head synthesis. By leveraging synthetic-to-real distillation and a residual editing approach for motion, it achieves a balance between realistic 3D geometry and fluid animation. Future work will focus on improving handling of extreme poses and enhancing background realism through advanced generative inpainting.

Summary updated

BibTeX
@inproceedings{ye2024real,
  author = {Zhenhui Ye and Tianyun Zhong and Yi Ren and Jiaqi Yang and Weichuang Li and Jiawei Huang and Ziyue Jiang and Jinzheng He and Rongjie Huang and Jinglin Liu and Chen Zhang and Xiang Yin and Zejun Ma and Zhou Zhao},
  title = {{Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=7ERQPyR2eb},
}
Figure