Abstract
Traditional 3D facial rigging is a labor-intensive process often constrained by specific mesh topologies. This paper presents Neural Face Rigging (NFR), an end-to-end deep learning framework designed to automatically rig and retarget 3D facial meshes regardless of their structure or origin. By leveraging a dual-input encoder and Neural Jacobian Fields, NFR achieves a triangulation-agnostic system that maintains high interpretability through a FACS-based latent space. Our results demonstrate a 32.3% reduction in error compared to state-of-the-art methods and show robust performance on "in-the-wild" scans.
Introduction
In the field of 3D animation, rigging—the process of creating a digital skeleton or control system for a mesh—remains a significant bottleneck. While automated rigging solutions have been proposed, they frequently struggle with two primary challenges: topology dependency and the tradeoff between realism and control. Most existing models require a fixed number of vertices and a specific triangulation pattern, making them brittle when applied to raw scans or meshes from diverse sources. Furthermore, while linear models like 3D Morphable Models (3DMMs) offer intuitive control, they often lack the non-linear realism of high-end character rigs.
Neural Face Rigging (NFR) addresses these limitations by providing a framework that is both triangulation-agnostic and interpretable. By combining synthetic data for structural guidance and real-world scans for surface realism, NFR allows for the seamless transfer of complex expressions between disparate meshes using an artist-friendly control system.
Methodology
The NFR architecture is built upon a specialized autoencoder designed to disentangle identity from expression while remaining flexible to varying mesh geometries.
Dual-Input Encoders
To process arbitrary meshes, NFR employs a two-pronged encoding strategy. First, it renders a 2D image of the face for processing via a standard Convolutional Neural Network (CNN). Second, it utilizes DiffusionNet, a state-of-the-art mesh encoder that operates on the surface geometry rather than a fixed vertex index. This combination allows the system to extract features from any 3D face, regardless of its underlying "connect-the-dots" pattern.
Interpretable Latent Space
A core innovation of NFR is its structured latent space, which is divided into two distinct components:
- $z_{FACS}$: A 53-dimensional space mapped directly to the Facial Action Coding System (FACS). This provides animators with familiar sliders for specific muscle movements, such as "jaw open" or "brow lowerer."
- $z_{ext}$: An "extended" latent space that captures fine-grained, non-linear details—such as skin wrinkles and subtle folds—that are typically omitted in standard linear FACS rigs.
Neural Jacobian Fields (NJF)
The decoder utilizes Neural Jacobian Fields to translate latent codes back into 3D deformations. Rather than predicting absolute vertex positions, the decoder predicts how each triangle in the mesh should deform (the Jacobian). This approach ensures that the model can be applied to meshes with any polygon count or triangulation, facilitating true "plug-and-play" functionality.
Training and Datasets
The model’s performance is rooted in a multi-dataset training scheme that balances interpretability with realism.
- ICT FaceKit: This synthetic dataset, based on a linear 3DMM, was used to supervise the 53 FACS parameters, establishing the semantic meaning of the control sliders.
- Multiface: A high-quality dataset of 4D scans of real individuals was used to fine-tune the model, teaching it how real skin moves and interacts non-linearly.
- Evaluation: The system was validated against various "in-the-wild" sources, including MetaHuman, FLAME, FaceWarehouse, and noisy raw scans.
Key Insights
The success of NFR stems from several key design choices. By initializing the model on synthetic data and fine-tuning on real scans, the authors achieved "supervised interpretability" without requiring ground-truth FACS labels for real-world data. Additionally, because NFR is a neural network, it naturally learns the complex, non-linear interactions of facial muscles. This prevents the "exploded" or unnatural look often seen in linear rigs when multiple expressions are combined. Finally, by prioritizing surface geometry over specific vertex topology, NFR achieves a level of robustness previously unseen in automated rigging.
Results
NFR demonstrates significant improvements over existing state-of-the-art methods like COMA and SpiralNet++. Quantitative analysis on the Multiface dataset revealed a 32.3% reduction in error at the 90th percentile. Qualitatively, the model proves highly robust, capable of retargeting expressions from high-fidelity meshes onto noisy, incomplete scans with holes and missing data while preserving the character of the expression. Furthermore, the system maintains artist control, allowing users to manually adjust FACS sliders to refine automated retargeting results.
Conclusion and Future Work
Neural Face Rigging represents a significant step toward automating the character animation pipeline. While the current model requires a manual "standardization" step to remove internal structures like the inner mouth, future iterations may incorporate automated semantic segmentation to streamline this process. Additionally, the integration of appearance models (textures and lighting) remains a promising avenue for achieving fully photo-realistic neural animation.
Summary updated