Riggable 3D Face Reconstruction via In-Network Optimization

Ziqian Bai, Zhaopeng Cui, Xiaoming Liu, Ping Tan

Abstract

This paper presents a method for riggable 3D face reconstruction from monocular images, which jointly estimates a personalized face rig and per-image parameters including expressions, poses, and illuminations. To achieve this goal, we design an end-to-end trainable network embedded with a differentiable in-network optimization. The network first parameterizes the face rig as a compact latent code with a neural decoder, and then estimates the latent code as well as per-image parameters via a learnable optimization. By estimating a personalized face rig, our method goes beyond static reconstructions and enables downstream applications such as video retargeting. In-network optimization explicitly enforces constraints derived from the first principles, thus introduces additional priors than regression-based methods. Finally, data-driven priors from deep learning are utilized to constrain the ill-posed monocular setting and ease the optimization difficulty. Experiments demonstrate that our method achieves SOTA reconstruction accuracy, reasonable robustness and generalization ability, and supports standard face rig applications.

Introduction

Monocular 3D face reconstruction is a critical task for applications in AR/VR, gaming, and digital telepresence. While existing methods have made significant strides in capturing facial geometry from a single image, they often produce "static" reconstructions that lack the flexibility required for animation. Creating a "riggable" 3D face model—a parametric digital puppet that can be driven by new expressions—traditionally requires expensive multi-camera setups or 3D scanners.

Current monocular approaches generally fall into two categories: traditional optimization and deep regression. Traditional optimization is accurate but computationally expensive and sensitive to initialization. Deep regression is fast but often lacks geometric precision because it fails to explicitly enforce physical constraints like multi-view consistency. To address these challenges, we introduce INORig, a framework that embeds differentiable optimization directly into a neural network to achieve high-fidelity, riggable reconstructions.

The INORig Framework

The core of our approach is In-Network Optimization. By integrating a differentiable optimization module within the inference process, the system can iteratively refine the face model, combining the robustness of deep learning with the mathematical rigor of geometric first principles.

Overview of the INORig pipeline, demonstrating 3D face reconstruction from a source image and subsequent expression retargeting to different target identities.

The reconstruction process follows a three-level coarse-to-fine scheme, gradually adding detail to both geometry and albedo. A Neural Rig Decoder parameterizes the face rig as a compact latent "rig code." This decoder is conditioned on input images, allowing it to capture person-specific facial structures. During inference, the system minimizes an energy function that accounts for landmark alignment, photometric consistency, and multi-view appearance consistency. This optimization is handled by a differentiable solver using learned step sizes, ensuring both speed and stability.

Experimental Results

We evaluated INORig on several standard datasets, including BU3DFE, Bosphorus, and the NoW dataset. Our method achieves state-of-the-art (SOTA) accuracy, outperforming previous regression-based models. For instance, on the BU3DFE dataset, INORig achieved a mean geometric error of 1.00mm, compared to approximately 1.38mm for existing regression methods.

Beyond quantitative metrics, INORig excels in capturing personalized traits, such as unique chin shapes or the specific folds of a smile. The resulting models are fully riggable, enabling high-fidelity video retargeting. As shown in our experiments, expressions from a source actor can be seamlessly mapped onto a reconstructed rig of a different target identity while maintaining the target's unique characteristics.

Discussion and Conclusion

While INORig represents a significant advancement, it has limitations. The model currently struggles with extreme or unusual expressions and subtle eyelid movements, primarily due to the limited diversity of expressions in the training data. Additionally, textures generated from "in-the-wild" images with harsh lighting can sometimes exhibit artifacts.

Future work will focus on training with more diverse, realistically rendered datasets and exploring self-supervised learning on unconstrained video data to further enhance robustness. In summary, INORig provides a powerful bridge between deep learning and traditional optimization, enabling the creation of high-quality, animatable 3D face rigs from standard monocular input.

Summary updated

Abstract

This paper presents a method for riggable 3D face reconstruction from monocular images, which jointly estimates a personalized face rig and per-image parameters including expressions, poses, and illuminations. To achieve this goal, we design an end-to-end trainable network embedded with a differentiable in-network optimization. The network first parameterizes the face rig as a compact latent code with a neural decoder, and then estimates the latent code as well as per-image parameters via a learnable optimization. By estimating a personalized face rig, our method goes beyond static reconstructions and enables downstream applications such as video retargeting. In-network optimization explicitly enforces constraints derived from the first principles, thus introduces additional priors than regression-based methods. Finally, data-driven priors from deep learning are utilized to constrain the ill-posed monocular setting and ease the optimization difficulty. Experiments demonstrate that our method achieves SOTA reconstruction accuracy, reasonable robustness and generalization ability, and supports standard face rig applications.
BibTeX
@inproceedings{bai2021riggable,
  author = {Ziqian Bai and Zhaopeng Cui and Xiaoming Liu and Ping Tan},
  title = {{Riggable 3D Face Reconstruction via In-Network Optimization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00615},
}
Figure