Implicit Neural Head Synthesis via Controllable Local Deformation Fields

Chuhan Chen, Matthew O'Toole, Gaurav Bharaj, Pablo GarridoCVPR 2023 · DOI · OpenReview

Introduction

Creating realistic, animatable 3D digital doubles from a single camera remains a significant challenge in computer graphics. Traditional approaches like 3D Morphable Models (3DMMs) provide ease of control but often lack fine-grained details, such as skin pores or the complex geometry of the mouth interior. Conversely, newer Neural Radiance Fields (NeRFs) offer photorealistic results but are difficult to "rig" for animation. Most neural head models use a global deformation field to map observed frames to a static canonical space, which often leads to "entanglement"—where moving the mouth might inadvertently cause the eyes to twitch—and struggles with asymmetric expressions.

Methodology

This research introduces a part-based implicit representation that replaces a single global deformation field with an ensemble of Local Deformation Fields ($D_l$). The face is divided into regions centered around 34 specific landmarks, with each landmark controlled by its own small Multi-Layer Perceptron (MLP) that handles movement only in its immediate vicinity.

The proposed pipeline: Monocular frames are processed via 3D face tracking to define local deformation fields, which map observed coordinates to a canonical TensoRF space.

To ensure precision and avoid artifacts, the authors implement two key strategies:

  1. Attention Masks: Pre-computed masks prune redundant expression parameters for each local field, forcing the model to learn specialized, sparse movements and preventing "ghosting" across different facial regions.
  2. Local Control Loss: A novel loss function forces specific landmarks to map to consistent 3D coordinates in the canonical space, regardless of pose or expression.

The system utilizes TensoRF as the underlying canonical representation, providing a faster and more memory-efficient backbone than standard NeRF implementations.

Experimental Setup

The model was evaluated using monocular video sequences of four subjects, including high-definition footage of former President Barack Obama and subjects from the IM Avatar benchmark. The datasets consisted of approximately 6,500 training frames and 1,500–2,000 test frames per subject.

Results and Discussion

The decomposition into local parts significantly increases the model's expressive capacity, allowing it to capture non-linear details like deep creases and sharp mouth interiors that global models typically smooth over. Quantitatively, the model outperformed state-of-the-art baselines on the LPIPS perceptual similarity metric across most subjects.

Qualitative comparison showing the proposed method's ability to reconstruct sharper details and more accurate expressions compared to existing neural head synthesis techniques.

Qualitatively, the method produces noticeably sharper results for teeth, tongues, and fine skin details. Furthermore, the local field approach enables precise "local edits," such as forcing a single eye to wink or a cheek to puff, even when such asymmetric movements were not explicitly present in the training data.

Conclusion and Future Work

By bridging the gap between traditional rigging and modern neural rendering, this method provides a more intuitive and detailed framework for 3D head synthesis. While highly effective, the model currently faces limitations with extreme "out-of-distribution" poses and the modeling of non-facial parts like shoulders. Future work will focus on improving disentanglement in "in-the-wild" videos and extending the local field approach to full-body synthesis.

Summary updated

Abstract

High-quality reconstruction of controllable 3D head avatars from 2D videos is highly desirable for virtual human applications in movies, games, and telepresence. Neural implicit fields provide a powerful representation to model 3D head avatars with personalized shape, expressions, and facial parts, e.g., hair and mouth interior, that go beyond the linear 3D morphable model (3DMM). However, existing methods do not model faces with fine-scale facial features, or local control of facial parts that extrapolate asymmetric expressions from monocular videos. Further, most condition only on 3DMM parameters with poor(er) locality, and resolve local features with a global neural field. We build on part-based implicit shape models that decompose a global deformation field into local ones. Our novel formulation models multiple implicit deformation fields with local semantic rig-like control via 3DMM-based parameters, and representative facial landmarks. Further, we propose a local control loss and attention mask mechanism that promote sparsity of each learned deformation field. Our formulation renders sharper locally controllable nonlinear deformations than previous implicit monocular approaches, especially mouth interior, asymmetric expressions, and facial details.
BibTeX
@inproceedings{chen2023implicit,
  author = {Chuhan Chen and Matthew O'Toole and Gaurav Bharaj and Pablo Garrido},
  title = {{Implicit Neural Head Synthesis via Controllable Local Deformation Fields}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00048},
}
Figure