TL;DR
This paper introduces a generalized physical face model that can be automatically fitted to a single face image or 3D scan, enabling physics-based facial animation for any new identity. By learning from a large dataset without requiring explicit simulation during training, the model makes realistic, physically-accurate facial animation more accessible.
What Problem Does This Paper Address?
Creating realistic 3D facial animation has traditionally been a complex and labor-intensive process. While physically-based simulation offers superior realism and handles physical phenomena like self-collisions, it requires intricate setup for each character, including defining their anatomical "material space" and training identity-specific deformation models. This complexity has limited its widespread adoption in production pipelines. Existing methods often rely on blendshape models, which are simpler but lack physical accuracy and can lead to interpenetrations. This paper aims to bridge this gap by developing a generalized, physics-based facial model that is easy to fit and animate for any new identity, democratizing access to high-quality facial animation.
Methods
The core of this work is a novel neural network architecture that learns a generalized physical face model. Key innovations include:
- Simulation-Free Learning: The model is trained without requiring computationally expensive Finite Element Method (FEM) simulations in the loop. Instead, it learns to generate deformations that are compatible with a physics simulator by optimizing for physically-inspired loss functions (e.g., reconstruction loss on skin, rigidity loss for bones, fixation loss for the skull, and a soft loss for bio-mechanically plausible deformation).
- Material Space Morphing: To avoid the need for manual anatomical modeling for each identity, the model learns to automatically generate an identity-specific material space (bones, skin, and soft tissue) by morphing a single canonical material space. This is achieved using a neural network conditioned on an identity latent code.
- Implicit Neural Representation: The model uses implicit neural networks to represent deformations and material spaces, allowing for continuous and differentiable outputs.
- Latent Space Parameterization: Identity and expression latent codes are parameterized using a standard 3D Morphable Model (3DMM) to enable intuitive control and interpolation.
Datasets
The primary dataset used for training is a large 3D face dataset containing 336 identities with approximately 13,000 face scans across various expressions. This dataset primarily provides skin geometry. For bone shape constraints, the SCULPTOR parametric bone generator is used. For evaluation, a static test set of 28 unseen identities and a dynamic test set of 5 unseen performance sequences are used. The FACESCAPE dataset is also used for comparative evaluation.
Key Insights and Ideas
- Decoupling Simulation from Training: The most significant insight is that it's possible to train a physically-accurate facial model without running full physics simulations during training. This drastically reduces computational cost and enables training on large, diverse datasets.
- Generative Material Space: Learning to "morph" a canonical anatomical model into an identity-specific one is a powerful way to bypass manual modeling and achieve personalized anatomy automatically.
- Unified Framework for Physical Animation: The model bridges the gap between simple, artist-friendly models (like 3DMMs) and complex, physically-accurate simulations by providing a generalized, animatable physical model.
Key Results
- Accurate Fitting: The model can be fitted to unseen identities from single 3D scans or even single face images with high accuracy, comparable to existing 3DMMs.
- Physical Effects: The fitted models can be animated to produce realistic physical effects, including collision avoidance (lip-lip, tooth-lip), gravity, paralysis, and bone reshaping.
- Efficiency: The simulation-free training approach makes the model scalable to hundreds of identities, significantly faster than previous methods. Testing involves a quick latent code optimization and then simulation.
- Generalization: The model demonstrates good generalization to unseen identities and expressions, enabling animation retargeting and latent space interpolation for novel identity generation.
Limitations and Future Work
- Mouth Interior: The model does not accurately model the interior of the mouth (e.g., teeth) due to limitations in the training data.
- Biological Accuracy: While aiming for physical accuracy, the learned actuations are not guaranteed to be biologically precise for all unseen identities.
- Specificity vs. Generality: The model prioritizes generality, which might limit its ability to capture highly specific actor-like facial details.
- Non-Human Characters: The model struggles with non-human characters, indicating a reliance on human facial anatomy.
Future work could involve extending the model to include mouth interiors, improving biological accuracy, and exploring more fine-grained control over specific facial features.
Summary updated