TL;DR
HRAvatar introduces a novel 3D Gaussian Splatting framework to create high-fidelity, fully animatable, and realistically relightable 3D head avatars from a single monocular video. By combining an end-to-end expression encoder with learnable per-point deformations and physically-based shading, it achieves state-of-the-art visual quality and real-time rendering speeds (~155 FPS).
What Problem Does This Paper Address?
Creating realistic 3D head avatars from standard smartphone videos (monocular input) is a major goal for VR, gaming, and telepresence. While 3D Gaussian Splatting (3DGS) has made real-time rendering possible, existing methods fall short in three key areas:
- Rigid Deformations: They rig Gaussian points directly to standard parametric templates (like FLAME), failing to capture person-specific nuances, wrinkles, or complex hair movement.
- Tracking Errors: They rely on offline pre-tracking tools to estimate facial expressions, meaning any initial tracking errors permanently degrade the final avatar quality.
- Baked-in Lighting: They directly optimize the color of the Gaussians, baking the original environment's lighting into the avatar. This makes it impossible to realistically place the avatar into a new lighting environment (relighting).
Methods
HRAvatar tackles these issues with a three-pronged architecture:
- Precise Expression Tracking: Instead of relying solely on frozen offline tracking, the authors integrate a learnable expression encoder trained end-to-end via photometric loss. This dynamically refines expression and jaw parameters during avatar reconstruction.
- Geometry Deformation Modeling: To move beyond the constraints of rigid parametric models, HRAvatar introduces learnable linear blendshapes and learnable linear blend skinning (LBS). Each Gaussian point independently learns its own shape, expression, and pose bases, alongside personalized skinning weights, allowing highly flexible and personalized facial movements.
- Appearance Modeling: To enable true relighting, the framework discards traditional spherical harmonics and decomposes head appearance into physically-based material properties: albedo, roughness, and Fresnel base reflectance. It uses a Disney BRDF model with a SplitSum approximation for efficient, real-time shading under arbitrary environment maps. To prevent lighting highlights from corrupting the albedo channel, they introduce an albedo pseudo-prior loss.
Datasets
The method was evaluated across three distinct datasets:
- INSTA dataset: 10 subjects with pre-cropped and segmented images.
- HDTF dataset: 8 subjects collected from the internet to test robustness on in-the-wild videos.
- Self-captured dataset: 5 subjects recorded using standard mobile phones to demonstrate real-world applicability.
Key Insights and Ideas
- End-to-End Tracking Refinement is Crucial: Traditional pipelines treat face tracking as a solved preprocessing step. HRAvatar shows that optimizing expression parameters jointly with the 3D representation dramatically cleans up artifacts and improves fine-grained details like mouth shapes and blinking.
- Per-Point Learnable Bases Overcome Template Rigidity: By letting individual Gaussians learn their own deformation behavior rather than strictly following a mesh template, the model can capture complex personalized geometry (like teeth details and fine wrinkles) that standard FLAME models miss.
- Decoupling Material from Lighting: Using an explicit albedo pseudo-prior prevents the optimization from "cheating" by baking shadows or highlights into the base texture, which is the secret sauce behind its clean relighting performance.
Key Results
- State-of-the-Art Quality: HRAvatar outperforms existing methods (including INSTA, Flash-Avatar, and GBS) across all quantitative metrics. For instance, on the INSTA dataset, it achieves a PSNR of 30.36 (compared to GBS's 29.64) and a significantly lower LPIPS score of 0.0569 (indicating much sharper, more realistic details).
- Real-Time Performance: Despite the added complexity of physically-based shading and dynamic deformations, HRAvatar renders at ~155 FPS on a single NVIDIA 3090 GPU, making it fully viable for interactive applications.
- Versatile Applications: The model demonstrates excellent visual fidelity in self-reenactment, cross-reenactment, novel view synthesis, and dynamic relighting under rotating environment maps. It also supports simple material editing (e.g., increasing Fresnel reflectance to create a metallic effect).
Limitations and Future Work
- Data Dependency: When training data is sparse, the model still relies heavily on FLAME priors, which can limit its ability to reconstruct complex hair or external accessories accurately.
- Material Entanglement: Despite the albedo prior, some fine shadows or wrinkles can still occasionally be mis-coupled into the albedo or reflectance channels, leading to minor artifacts under extreme specular reflections.
- Monocular View Constraints: Reconstructing the complete back of the head remains unfeasible from a pure frontal monocular video because face tracking becomes highly unreliable as the head rotation approaches a 90-degree profile view.
- Future Directions: The authors suggest integrating semantic information to better constrain material learning (e.g., ensuring hair and skin maintain consistent material properties) and offloading more computation to the GPU via custom CUDA kernels to further boost performance.
Summary updated