Introduction
Creating realistic, animatable 3D digital doubles from a single camera remains a significant challenge in computer graphics. Traditional approaches like 3D Morphable Models (3DMMs) provide ease of control but often lack fine-grained details, such as skin pores or the complex geometry of the mouth interior. Conversely, newer Neural Radiance Fields (NeRFs) offer photorealistic results but are difficult to "rig" for animation. Most neural head models use a global deformation field to map observed frames to a static canonical space, which often leads to "entanglement"—where moving the mouth might inadvertently cause the eyes to twitch—and struggles with asymmetric expressions.
Methodology
This research introduces a part-based implicit representation that replaces a single global deformation field with an ensemble of Local Deformation Fields ($D_l$). The face is divided into regions centered around 34 specific landmarks, with each landmark controlled by its own small Multi-Layer Perceptron (MLP) that handles movement only in its immediate vicinity.
To ensure precision and avoid artifacts, the authors implement two key strategies:
- Attention Masks: Pre-computed masks prune redundant expression parameters for each local field, forcing the model to learn specialized, sparse movements and preventing "ghosting" across different facial regions.
- Local Control Loss: A novel loss function forces specific landmarks to map to consistent 3D coordinates in the canonical space, regardless of pose or expression.
The system utilizes TensoRF as the underlying canonical representation, providing a faster and more memory-efficient backbone than standard NeRF implementations.
Experimental Setup
The model was evaluated using monocular video sequences of four subjects, including high-definition footage of former President Barack Obama and subjects from the IM Avatar benchmark. The datasets consisted of approximately 6,500 training frames and 1,500–2,000 test frames per subject.
Results and Discussion
The decomposition into local parts significantly increases the model's expressive capacity, allowing it to capture non-linear details like deep creases and sharp mouth interiors that global models typically smooth over. Quantitatively, the model outperformed state-of-the-art baselines on the LPIPS perceptual similarity metric across most subjects.
Qualitatively, the method produces noticeably sharper results for teeth, tongues, and fine skin details. Furthermore, the local field approach enables precise "local edits," such as forcing a single eye to wink or a cheek to puff, even when such asymmetric movements were not explicitly present in the training data.
Conclusion and Future Work
By bridging the gap between traditional rigging and modern neural rendering, this method provides a more intuitive and detailed framework for 3D head synthesis. While highly effective, the model currently faces limitations with extreme "out-of-distribution" poses and the modeling of non-facial parts like shoulders. Future work will focus on improving disentanglement in "in-the-wild" videos and extending the local field approach to full-body synthesis.
Summary updated