Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis
Introduction
Generating realistic talking head videos from a single source image—the "one-shot" setting—is a significant challenge in computer vision. While 2D-based methods can produce high-quality frames, they often fail to maintain 3D consistency, leading to warping artifacts during head rotations. Conversely, existing 3D methods like Neural Radiance Fields (NeRF) typically require extensive per-subject training. Real3D-Portrait addresses these limitations by introducing a framework that lifts a single 2D image into a 3D representation capable of stable, high-fidelity animation driven by either video or audio.
Methodology
The Real3D-Portrait framework is a multi-stage pipeline designed for robust 3D reconstruction and realistic motion synthesis. It decomposes the task into specialized modules for geometry, motion, and rendering.
1. Image-to-Plane (I2P) Model
The core of the system is the I2P model, which converts a 2D source image into a 3D tri-plane representation. It employs a dual-branch architecture: a Vision Transformer (ViT) branch to capture global 3D structure and a VGG branch for fine-grained texture details. To ensure strong geometric priors, the model is pre-trained on synthetic data from a 3D GAN (EG3D), allowing it to generalize to unseen real-world faces.
2. Motion Adapter (MA)
To animate the 3D head, the Motion Adapter predicts a "residual motion diff-plane." Instead of complex 3D deformations, it uses a Projected Normalized Coordinate Code (PNCC) to represent facial expressions. The MA calculates the necessary updates to the canonical tri-plane, ensuring stable animation and superior identity preservation even during rapid movements.
3. Head-Torso-Background (HTB-SR) Model
To create a complete portrait, the system separately models the head, torso, and background. The head is rendered via a 3D volume renderer, while the torso is handled through 2D warping. A KNN-based inpainting module manages the background. These components are fused using alpha-blending and refined by a super-resolution (SR) model to produce a seamless, high-resolution output.
4. Audio-to-Motion (A2M)
For audio-driven synthesis, a VAE-based model converts HuBERT-processed audio into 3DMM expression parameters. These parameters are then mapped to the PNCC format required by the Motion Adapter.
Experiments and Results
The model was trained on the CelebV-HQ and VoxCeleb2 datasets. Quantitative evaluations showed that Real3D-Portrait outperforms state-of-the-art 2D (e.g., Face-vid2vid) and 3D (e.g., HiDe-NeRF) baselines in terms of image quality (FID), identity similarity (CSIM), and lip-sync accuracy (Sync-score). Qualitative results demonstrate the model's ability to maintain structural integrity at large rotation angles, a common failure point for previous methods. A user study further confirmed its superiority in visual quality and synchronization.
Conclusion
Real3D-Portrait presents a robust solution for one-shot 3D talking head synthesis. By leveraging synthetic-to-real distillation and a residual editing approach for motion, it achieves a balance between realistic 3D geometry and fluid animation. Future work will focus on improving handling of extreme poses and enhancing background realism through advanced generative inpainting.
Summary updated