Riggable 3D Face Reconstruction via In-Network Optimization

Ziqian Bai, Zhaopeng Cui, Xiaoming Liu, Ping Tan

Abstract

Capturing personalized, animatable 3D face rigs from monocular images remains a significant challenge in computer vision. This paper presents a novel framework that embeds a differentiable optimization process directly within a deep neural network. By jointly learning a person-specific "rig code" and per-image parameters such as pose, expression, and lighting, the proposed method achieves state-of-the-art reconstruction accuracy and enables high-fidelity facial retargeting.

Introduction

The reconstruction of 3D facial geometry from 2D images is a classic ill-posed problem. While static reconstruction has seen significant progress, many applications—including AR/VR, gaming, and telepresence—require "riggable" models. These are personalized parametric models that can be driven by new expressions while maintaining the subject's unique identity. Traditional methods for creating such rigs often require expensive hardware or rely on rigid, low-dimensional models like 3D Morphable Models (3DMMs), which frequently fail to capture individual nuances or generalize to unconstrained "in-the-wild" photos.

Methodology

The proposed approach bridges the gap between the representational power of deep learning and the physical constraints of classical optimization through three key components:

Neural Rig Decoder

Unlike fixed linear models, the framework utilizes a neural decoder. This decoder accepts a compact "rig code" ($\alpha$) representing identity and an expression parameter ($\beta$) to generate a 3D mesh. This architecture allows the model to learn non-linear, person-specific details, such as unique wrinkles and bone structures, that standard 3DMMs typically miss.

In-Network Optimization

The core innovation lies in embedding a differentiable optimization loop within the network. Instead of a single-pass regression, the system iteratively refines the rig code and per-image parameters by minimizing an energy function. This function incorporates landmark alignment, photometric consistency, and multi-view appearance consistency to ensure the person's identity remains stable across different photos. Notably, the network learns optimal step sizes for these updates, facilitating faster and more stable convergence compared to standard gradient descent.

Coarse-to-Fine Scheme

To ensure robustness, the optimization occurs across three levels of increasing resolution. This allows the model to first establish the basic head shape before refining fine-grained geometric details.

Key Results and Evaluation

The model was trained on the Stirling/ESRC 3D face database and evaluated against benchmarks like BU3DFE, Bosphorus, and the "in-the-wild" NoW dataset.

Accuracy and Personalization

On the BU3DFE dataset, the method achieved a mean geometric error of 1.00mm, outperforming both multi-view (e.g., Bai20 at 1.11mm) and single-view (e.g., Tewari18 at 1.78mm) baselines. Qualitative analysis confirms that the model captures individual traits—such as specific chin shapes or cheek folds—more faithfully than generic 3DMM-based approaches.

Video Retargeting

A primary strength of the riggable output is its application in video retargeting. By mapping the expressions of a source actor onto the reconstructed rig of a target subject, the system produces convincing animations that preserve the target's identity while mimicking the source's performance.

Figure 1: Overview of the reconstruction and retargeting pipeline. The system takes source expressions and applies them to personalized rigs (Target A and B) created via the in-network optimization process.

Discussion and Limitations

While the method represents a significant advancement, certain challenges remain. The model can struggle with extreme facial contortions that deviate significantly from the training distribution. Additionally, very fine-grained motions, such as subtle eyelid movements, and handling heavy occlusions (e.g., hands) remain areas for improvement. Future research may focus on incorporating temporal data from video sequences to further enhance the dynamic fidelity of the reconstructed rigs.

Summary updated

Abstract

This paper presents a method for riggable 3D face reconstruction from monocular images, which jointly estimates a personalized face rig and per-image parameters including expressions, poses, and illuminations. To achieve this goal, we design an end-to-end trainable network embedded with a differentiable in-network optimization. The network first parameterizes the face rig as a compact latent code with a neural decoder, and then estimates the latent code as well as per-image parameters via a learnable optimization. By estimating a personalized face rig, our method goes beyond static reconstructions and enables downstream applications such as video retargeting. In-network optimization explicitly enforces constraints derived from the first principles, thus introduces additional priors than regression-based methods. Finally, data-driven priors from deep learning are utilized to constrain the ill-posed monocular setting and ease the optimization difficulty. Experiments demonstrate that our method achieves SOTA reconstruction accuracy, reasonable robustness and generalization ability, and supports standard face rig applications.
BibTeX
@inproceedings{bai2021riggable,
  author = {Ziqian Bai and Zhaopeng Cui and Xiaoming Liu and Ping Tan},
  title = {{Riggable 3D Face Reconstruction via In-Network Optimization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00615},
}
Figure