TL;DR
3DRealHead creates highly detailed, animatable 3D head avatars from just 1 to 3 images. By combining standard 3D morphable models with fine-grained mouth-region image gradients extracted from a driving webcam video, it captures realistic expressions and person-specific details like teeth without requiring dense, per-person training videos.
What Problem Does This Paper Address?
Creating realistic 3D head avatars usually requires a steep trade-off. You either need a multi-view studio setup with dense video data (which is expensive and inaccessible to everyday users), or you rely on few-shot methods that use 3D Morphable Models (3DMMs).
However, standard 3DMMs operate in a low-dimensional expression space. This means they completely miss highly person-specific, fine-grained details—most notably the mouth interior, tongue, and teeth. As a result, avatars driven by a standard webcam often look uncanny or generic when talking. 3DRealHead tackles this challenge, aiming to reconstruct high-fidelity, expressive avatars from only a few images while faithfully capturing complex mouth dynamics during animation.
Methods
The authors introduce a framework that combines a learned 3D human head prior with a hybrid expression control signal:
- Model Backbone: The core architecture is a Style U-Net that predicts 3D Gaussian primitives in UV-space relative to a FLAME head mesh template.
- Hybrid Expression Conditioning: To animate the avatar, the expression encoder receives a two-part signal from the driving video: coarse 3DMM position offsets (from the FLAME model) and 2D image gradient maps extracted specifically from the driving actor's mouth region. This gradient map bypasses the limitations of the 3DMM expression space, capturing fine-grained structural details of the mouth interior.
- Few-Shot Inversion: To personalize the avatar, the user provides 1–3 neutral images. The system runs a two-stage inversion process, optimizing the hierarchical feature maps of the identity encoder and the final layer of the decoder while keeping the rest of the network frozen.
- Animation and Rendering: The predicted 3D Gaussians are deformed via Linear Blend Skinning (LBS) based on the driving video's FLAME tracking and rendered in real-time using 3D Gaussian Splatting (3DGS).
Datasets
- NeRSemble: Used to train the underlying 3D prior. It contains high-quality studio data of 200 real subjects, each captured across 16 camera views with 6 expression sequences.
- INSTA ("In-the-Wild"): Used to evaluate how well the model generalizes to everyday settings. This dataset consists of monocular, frontal talking videos recorded with a standard camera.
Key Insights and Ideas
- Gradients Over RGB for Cross-Identity Robustness: Instead of feeding raw RGB pixels of the driving mouth into the encoder, the authors use image gradients. This prevents the model from accidentally copying the driving actor's skin tone or lighting conditions onto the target avatar, making expression transfer robust to illumination changes.
- Multi-Resolution Inversion: Rather than optimizing a single global latent vector during personalization, the method optimizes hierarchical feature maps across multiple resolutions. This preserves the structural prior of the decoder while allowing the model to inject sharp, high-fidelity identity details.
- 3DMMs as Anchors, Not Cages: Treating the 3DMM as an underlying geometric anchor for canvas deformation—while letting 3D Gaussians dynamically handle fine-grained appearance via the U-Net—yields stable global tracking alongside unconstrained local expressivity.
Key Results
- Studio Data (1-Shot): 3DRealHead outperforms state-of-the-art one-shot prior-based methods (like GAGAvatar, GPAvatar, and LAM) across all major metrics, achieving a higher PSNR (33.15) and lower LPIPS (0.036).
- In-the-Wild Data: When tested on casual webcam-style data, it outperforms synthetic-prior methods like SynShot by a considerable margin.
- Efficiency vs. Video-Based Methods: Remarkably, 3DRealHead approaches the visual quality of heavy monocular video methods (such as INSTA and FlashAvatar) that require thousands of frames and minutes of per-subject training, despite only using 1–3 frames for enrollment.
- Speed: The entire driving and rendering pipeline runs at a smooth 27 FPS, making it viable for real-time applications.
Limitations and Future Work
- Identity Leakage: In cross-reenactment scenarios (driving Person A's avatar with Person B's video), some of the driving person's distinctive mouth structure can leak into the target avatar due to the direct gradient conditioning.
- Forehead Wrinkles: Because the forehead region relies purely on standard 3DMM tracking, it can suffer from reduced or missing dynamic wrinkles. The authors suggest extending their gradient-conditioning technique to the forehead in future iterations.
- Extreme Views: The model cannot handle extreme side-view driving videos well and can exhibit color shifts on out-of-distribution data. Training on a prior dataset with even greater camera and appearance variety could help resolve this.
Summary updated