Learning a Generalized Physical Face Model From Data

Lingchen Yang, Gaspard Zoss, Prashanth Chandran, Markus Gross, Barbara Solenthaler, Eftychios Sifakis, Derek BradleyACM Trans. Graphics 2024 · DOI · OpenReview

TL;DR

This paper introduces a generalized physical face model that can be automatically fitted to a single face image or 3D scan, enabling physics-based facial animation for any new identity. By learning from a large dataset without requiring explicit simulation during training, the model makes realistic, physically-accurate facial animation more accessible.

What Problem Does This Paper Address?

Creating realistic 3D facial animation has traditionally been a complex and labor-intensive process. While physically-based simulation offers superior realism and handles physical phenomena like self-collisions, it requires intricate setup for each character, including defining their anatomical "material space" and training identity-specific deformation models. This complexity has limited its widespread adoption in production pipelines. Existing methods often rely on blendshape models, which are simpler but lack physical accuracy and can lead to interpenetrations. This paper aims to bridge this gap by developing a generalized, physics-based facial model that is easy to fit and animate for any new identity, democratizing access to high-quality facial animation.

Methods

The core of this work is a novel neural network architecture that learns a generalized physical face model. Key innovations include:

  • Simulation-Free Learning: The model is trained without requiring computationally expensive Finite Element Method (FEM) simulations in the loop. Instead, it learns to generate deformations that are compatible with a physics simulator by optimizing for physically-inspired loss functions (e.g., reconstruction loss on skin, rigidity loss for bones, fixation loss for the skull, and a soft loss for bio-mechanically plausible deformation).
  • Material Space Morphing: To avoid the need for manual anatomical modeling for each identity, the model learns to automatically generate an identity-specific material space (bones, skin, and soft tissue) by morphing a single canonical material space. This is achieved using a neural network conditioned on an identity latent code.
  • Implicit Neural Representation: The model uses implicit neural networks to represent deformations and material spaces, allowing for continuous and differentiable outputs.
  • Latent Space Parameterization: Identity and expression latent codes are parameterized using a standard 3D Morphable Model (3DMM) to enable intuitive control and interpolation.

Datasets

The primary dataset used for training is a large 3D face dataset containing 336 identities with approximately 13,000 face scans across various expressions. This dataset primarily provides skin geometry. For bone shape constraints, the SCULPTOR parametric bone generator is used. For evaluation, a static test set of 28 unseen identities and a dynamic test set of 5 unseen performance sequences are used. The FACESCAPE dataset is also used for comparative evaluation.

Key Insights and Ideas

  • Decoupling Simulation from Training: The most significant insight is that it's possible to train a physically-accurate facial model without running full physics simulations during training. This drastically reduces computational cost and enables training on large, diverse datasets.
  • Generative Material Space: Learning to "morph" a canonical anatomical model into an identity-specific one is a powerful way to bypass manual modeling and achieve personalized anatomy automatically.
  • Unified Framework for Physical Animation: The model bridges the gap between simple, artist-friendly models (like 3DMMs) and complex, physically-accurate simulations by providing a generalized, animatable physical model.

Key Results

  • Accurate Fitting: The model can be fitted to unseen identities from single 3D scans or even single face images with high accuracy, comparable to existing 3DMMs.
  • Physical Effects: The fitted models can be animated to produce realistic physical effects, including collision avoidance (lip-lip, tooth-lip), gravity, paralysis, and bone reshaping.
  • Efficiency: The simulation-free training approach makes the model scalable to hundreds of identities, significantly faster than previous methods. Testing involves a quick latent code optimization and then simulation.
  • Generalization: The model demonstrates good generalization to unseen identities and expressions, enabling animation retargeting and latent space interpolation for novel identity generation.

Limitations and Future Work

  • Mouth Interior: The model does not accurately model the interior of the mouth (e.g., teeth) due to limitations in the training data.
  • Biological Accuracy: While aiming for physical accuracy, the learned actuations are not guaranteed to be biologically precise for all unseen identities.
  • Specificity vs. Generality: The model prioritizes generality, which might limit its ability to capture highly specific actor-like facial details.
  • Non-Human Characters: The model struggles with non-human characters, indicating a reliance on human facial anatomy.

Future work could involve extending the model to include mouth interiors, improving biological accuracy, and exploring more fine-grained control over specific facial features.

Summary updated

Abstract

Physically-based simulation is a powerful approach for 3D facial animation as the resulting deformations are governed by physical constraints, allowing to easily resolve self-collisions, respond to external forces and perform realistic anatomy edits. Today's methods are data-driven, where the actuations for finite elements are inferred from captured skin geometry. Unfortunately, these approaches have not been widely adopted due to the complexity of initializing the material space and learning the deformation model for each character separately, which often requires a skilled artist followed by lengthy network training. In this work, we aim to make physics-based facial animation more accessible by proposing a generalized physical face model that we learn from a large 3D face dataset. Once trained, our model can be quickly fit to any unseen identity and produce a ready-to-animate physical face model automatically. Fitting is as easy as providing a single 3D face scan, or even a single face image. After fitting, we offer intuitive animation controls, as well as the ability to retarget animations across characters. All the while, the resulting animations allow for physical effects like collision avoidance, gravity, paralysis, bone reshaping and more.
BibTeX
@article{yang2024learning,
  author = {Lingchen Yang and Gaspard Zoss and Prashanth Chandran and Markus Gross and Barbara Solenthaler and Eftychios Sifakis and Derek Bradley},
  title = {{Learning a Generalized Physical Face Model From Data}},
  journal = {ACM Trans. Graphics},
  year = {2024},
  doi = {10.1145/3658189},
}
Figure