3DRealHead: Few-Shot Detailed Head Avatar

Jalees Nehvi, Timo Bolkart, Thabo Beeler, Justus Thies

TL;DR

3DRealHead creates highly detailed, animatable 3D head avatars from just 1 to 3 images. By combining standard 3D morphable models with fine-grained mouth-region image gradients extracted from a driving webcam video, it captures realistic expressions and person-specific details like teeth without requiring dense, per-person training videos.

What Problem Does This Paper Address?

Creating realistic 3D head avatars usually requires a steep trade-off. You either need a multi-view studio setup with dense video data (which is expensive and inaccessible to everyday users), or you rely on few-shot methods that use 3D Morphable Models (3DMMs).

However, standard 3DMMs operate in a low-dimensional expression space. This means they completely miss highly person-specific, fine-grained details—most notably the mouth interior, tongue, and teeth. As a result, avatars driven by a standard webcam often look uncanny or generic when talking. 3DRealHead tackles this challenge, aiming to reconstruct high-fidelity, expressive avatars from only a few images while faithfully capturing complex mouth dynamics during animation.

Methods

The authors introduce a framework that combines a learned 3D human head prior with a hybrid expression control signal:

  1. Model Backbone: The core architecture is a Style U-Net that predicts 3D Gaussian primitives in UV-space relative to a FLAME head mesh template.
  2. Hybrid Expression Conditioning: To animate the avatar, the expression encoder receives a two-part signal from the driving video: coarse 3DMM position offsets (from the FLAME model) and 2D image gradient maps extracted specifically from the driving actor's mouth region. This gradient map bypasses the limitations of the 3DMM expression space, capturing fine-grained structural details of the mouth interior.
  3. Few-Shot Inversion: To personalize the avatar, the user provides 1–3 neutral images. The system runs a two-stage inversion process, optimizing the hierarchical feature maps of the identity encoder and the final layer of the decoder while keeping the rest of the network frozen.
  4. Animation and Rendering: The predicted 3D Gaussians are deformed via Linear Blend Skinning (LBS) based on the driving video's FLAME tracking and rendered in real-time using 3D Gaussian Splatting (3DGS).
Fig. 2: Overview of 3DRealHead. Based on few-shot input data, an RGB texture is generated which serves as identity conditioning to the Style U-Net which predicts 3D Gaussian primitives in UV-space relative to a FLAME head mesh. To control detailed facial expressions from a driving input video, we extract 3DMM-based position maps [35] and image gradient features which are projected to the UV texture space. From this control signal, the model learns how to produce dynamically changing 3D Gaussian primitives which can be rendered using 3D Gaussian splatting. The Style U-Net is trained on the NeRSemble [24] dataset, such that it can act as a prior for few-shot inversion, where only the identity encoder EId is finetuned. During the training of the prior, we employ several reconstruction losses, as well as a multi-head discriminator D for adversarial training.

Datasets

  • NeRSemble: Used to train the underlying 3D prior. It contains high-quality studio data of 200 real subjects, each captured across 16 camera views with 6 expression sequences.
  • INSTA ("In-the-Wild"): Used to evaluate how well the model generalizes to everyday settings. This dataset consists of monocular, frontal talking videos recorded with a standard camera.

Key Insights and Ideas

  • Gradients Over RGB for Cross-Identity Robustness: Instead of feeding raw RGB pixels of the driving mouth into the encoder, the authors use image gradients. This prevents the model from accidentally copying the driving actor's skin tone or lighting conditions onto the target avatar, making expression transfer robust to illumination changes.
  • Multi-Resolution Inversion: Rather than optimizing a single global latent vector during personalization, the method optimizes hierarchical feature maps across multiple resolutions. This preserves the structural prior of the decoder while allowing the model to inject sharp, high-fidelity identity details.
  • 3DMMs as Anchors, Not Cages: Treating the 3DMM as an underlying geometric anchor for canvas deformation—while letting 3D Gaussians dynamically handle fine-grained appearance via the U-Net—yields stable global tracking alongside unconstrained local expressivity.

Key Results

  • Studio Data (1-Shot): 3DRealHead outperforms state-of-the-art one-shot prior-based methods (like GAGAvatar, GPAvatar, and LAM) across all major metrics, achieving a higher PSNR (33.15) and lower LPIPS (0.036).
  • In-the-Wild Data: When tested on casual webcam-style data, it outperforms synthetic-prior methods like SynShot by a considerable margin.
  • Efficiency vs. Video-Based Methods: Remarkably, 3DRealHead approaches the visual quality of heavy monocular video methods (such as INSTA and FlashAvatar) that require thousands of frames and minutes of per-subject training, despite only using 1–3 frames for enrollment.
  • Speed: The entire driving and rendering pipeline runs at a smooth 27 FPS, making it viable for real-time applications.
Qualitative comparison against one-shot prior-based methods

Limitations and Future Work

  • Identity Leakage: In cross-reenactment scenarios (driving Person A's avatar with Person B's video), some of the driving person's distinctive mouth structure can leak into the target avatar due to the direct gradient conditioning.
  • Forehead Wrinkles: Because the forehead region relies purely on standard 3DMM tracking, it can suffer from reduced or missing dynamic wrinkles. The authors suggest extending their gradient-conditioning technique to the forehead in future iterations.
  • Extreme Views: The model cannot handle extreme side-view driving videos well and can exhibit color shifts on out-of-distribution data. Training on a prior dataset with even greater camera and appearance variety could help resolve this.

Summary updated

Abstract

The human face is central to communication. For immersive applications, the digital presence of a person should mirror the physical reality, capturing the users idiosyncrasies and detailed facial expressions. However, current 3D head avatar methods often struggle to faithfully reproduce the identity and facial expressions, despite having multi-view data or learned priors. Learning priors that capture the diversity of human appearances, especially, for regions with highly person-specific features, like the mouth and teeth region is challenging as the underlying training data is limited. In addition, many of the avatar methods are purely relying on 3D morphable model-based expression control which strongly limits expressivity. To address these challenges, we are introducing 3DRealHead, a few-shot head avatar reconstruction method with a novel expression control signal that is extracted from a monocular video stream of the subject. Specifically, the subject can take a few pictures of themselves, recover a 3D head avatar and drive it with a consumer-level webcam. The avatar reconstruction is enabled via a novel few-shot inversion process of a 3D human head prior which is represented as a Style U-Net that emits 3D Gaussian primitives which can be rendered under novel views. The prior is learned on the NeRSemble dataset. For animating the avatar, the U-Net is conditioned on 3DMM-based facial expression signals, as well as features of the mouth region extracted from the driving video. These additional mouth features allow us to recover facial expressions that cannot be represented by the 3DMM leading to a higher expressivity and closer resemblance to the physical reality.
BibTeX
@inproceedings{nehvi2026drealhead,
  author = {Jalees Nehvi and Timo Bolkart and Thabo Beeler and Justus Thies},
  title = {{3DRealHead: Few-Shot Detailed Head Avatar}},
  booktitle = {FG},
  year = {2026},
  doi = {10.1109/fg67764.2026.11556522},
}

Loading related content...

Figure