Abstract
Capturing personalized, animatable 3D face rigs from monocular images remains a significant challenge in computer vision. This paper presents a novel framework that embeds a differentiable optimization process directly within a deep neural network. By jointly learning a person-specific "rig code" and per-image parameters such as pose, expression, and lighting, the proposed method achieves state-of-the-art reconstruction accuracy and enables high-fidelity facial retargeting.
Introduction
The reconstruction of 3D facial geometry from 2D images is a classic ill-posed problem. While static reconstruction has seen significant progress, many applications—including AR/VR, gaming, and telepresence—require "riggable" models. These are personalized parametric models that can be driven by new expressions while maintaining the subject's unique identity. Traditional methods for creating such rigs often require expensive hardware or rely on rigid, low-dimensional models like 3D Morphable Models (3DMMs), which frequently fail to capture individual nuances or generalize to unconstrained "in-the-wild" photos.
Methodology
The proposed approach bridges the gap between the representational power of deep learning and the physical constraints of classical optimization through three key components:
Neural Rig Decoder
Unlike fixed linear models, the framework utilizes a neural decoder. This decoder accepts a compact "rig code" ($\alpha$) representing identity and an expression parameter ($\beta$) to generate a 3D mesh. This architecture allows the model to learn non-linear, person-specific details, such as unique wrinkles and bone structures, that standard 3DMMs typically miss.
In-Network Optimization
The core innovation lies in embedding a differentiable optimization loop within the network. Instead of a single-pass regression, the system iteratively refines the rig code and per-image parameters by minimizing an energy function. This function incorporates landmark alignment, photometric consistency, and multi-view appearance consistency to ensure the person's identity remains stable across different photos. Notably, the network learns optimal step sizes for these updates, facilitating faster and more stable convergence compared to standard gradient descent.
Coarse-to-Fine Scheme
To ensure robustness, the optimization occurs across three levels of increasing resolution. This allows the model to first establish the basic head shape before refining fine-grained geometric details.
Key Results and Evaluation
The model was trained on the Stirling/ESRC 3D face database and evaluated against benchmarks like BU3DFE, Bosphorus, and the "in-the-wild" NoW dataset.
Accuracy and Personalization
On the BU3DFE dataset, the method achieved a mean geometric error of 1.00mm, outperforming both multi-view (e.g., Bai20 at 1.11mm) and single-view (e.g., Tewari18 at 1.78mm) baselines. Qualitative analysis confirms that the model captures individual traits—such as specific chin shapes or cheek folds—more faithfully than generic 3DMM-based approaches.
Video Retargeting
A primary strength of the riggable output is its application in video retargeting. By mapping the expressions of a source actor onto the reconstructed rig of a target subject, the system produces convincing animations that preserve the target's identity while mimicking the source's performance.
Discussion and Limitations
While the method represents a significant advancement, certain challenges remain. The model can struggle with extreme facial contortions that deviate significantly from the training distribution. Additionally, very fine-grained motions, such as subtle eyelid movements, and handling heavy occlusions (e.g., hands) remain areas for improvement. Future research may focus on incorporating temporal data from video sequences to further enhance the dynamic fidelity of the reconstructed rigs.
Summary updated