SCULPTOR: Skeleton-Consistent Face Creation Using a Learned Parametric Generator

Zesong Qiu, Yuwei Li, Dongming He, Qixuan Zhang, Longwen Zhang, Yinghao Zhang, Jingya Wang, Lan Xu, Xudong Wang, Yuyao Zhang, Jingyi Yu

TL;DR

This paper introduces SCULPTOR, a novel system for generating realistic 3D human faces that explicitly models the underlying skeletal structure (skull) alongside facial geometry and appearance. By leveraging a new large-scale dataset of CT scans and 3D face scans, SCULPTOR enables more anatomically consistent and characteristic facial variations than previous methods, opening doors for applications like forensic reconstruction and character design.

What Problem Does This Paper Address?

Creating realistic and diverse 3D human faces is crucial for applications ranging from digital avatars to medical simulations. While existing methods have made significant strides in modeling external facial features, textures, and expressions, they often overlook the fundamental anatomical relationship between the skull and the face. This oversight can lead to anatomically implausible or unconvincing facial variations. The paper argues that facial appearance is biomechanically linked to skeletal structures, and that a consistent facial model should respect these underlying anatomical rules. The challenge is to develop a system that can generate faces with high anatomical correctness and visual realism, allowing for nuanced control over facial features that are rooted in skeletal variations.

Methods

SCULPTOR employs a hybrid parametric-physical representation to model 3D faces. The core of the system is a learned parametric generator that jointly models the skull, face geometry, and face appearance. This is achieved by:

  • Skeleton-Consistent Parametric Model: SCULPTOR extends existing parametric face models (like FLAME) by explicitly incorporating the skull's influence. It models face geometry as a combination of shape, trait, pose, and expression blend shapes, where the "trait" component specifically captures variations influenced by skeletal structure.
  • Linear Blend Skinning (LBS): The model uses LBS to deform the face mesh based on skeletal poses and learned skinning weights.
  • Trait Component: A novel parameter, the "trait component" ($\gamma$), is introduced to represent physiologically sound variations in the internal skeleton and their influence on face shape. This is learned from pre- and post-surgery CT data, capturing how skeletal changes affect external appearance.
  • Data-Driven Framework: The entire model is trained in a data-driven manner using a comprehensive dataset.
  • Registration and Parameter Learning: A multi-stage process involves registering template skull and face meshes to the dataset, followed by iterative learning of shape, trait, pose, and appearance parameters.

Datasets

  • LUCY Dataset: The paper introduces LUCY, a new large-scale dataset specifically designed for this research. It comprises high-quality Computed Tomography (CT) scans of complete human heads from 72 subjects (31 male, 41 female), with scans taken both before and after orthognathic surgeries. This dataset is crucial as it provides paired pre- and post-surgery data, allowing for the study of how skeletal changes influence facial appearance. It also includes 3D facial scans with texture maps.
  • FaceScape Dataset: Used for learning pose-related parameters (skinning weight and pose blend shapes).
  • 3DSCANSTORE: Used for additional face texture assets.
  • FFHQ Dataset: Used for training the skull inference model.

Key Insights and Ideas

  • Skull as the Foundation: The paper strongly emphasizes that the skull is the primary determinant of facial structure and appearance. Ignoring this underlying anatomy leads to less convincing and anatomically incorrect facial variations.
  • Trait Space for Nuance: The introduction of a "trait space" is a key innovation. This allows for controlling subtle, anatomically-grounded variations in facial features (like jaw width or length) that are directly linked to skeletal changes, going beyond global shape adjustments.
  • Bridging Medical Imaging and Graphics: The use of CT scans from plastic surgery provides a unique bridge between medical imaging data and computer graphics, enabling the learning of anatomically informed facial models.
  • Disentangling Factors: The model aims to disentangle shape, trait, pose, and expression, allowing for more controlled and interpretable editing of facial characteristics.

Key Results

  • Improved Skull Fitting: SCULPTOR-SIMPLE (without trait components) and SCULPTOR (with trait components) both showed improved skull fitting performance (lower Mean Vertex Squared Error) compared to baseline methods on test data. SCULPTOR with trait components generally outperformed SCULPTOR-SIMPLE.
  • Realistic Facial Variations: Qualitative results demonstrate SCULPTOR's ability to generate diverse and characteristic facial variations, including those influenced by skeletal changes (e.g., jaw shape, cheekbone height).
  • Effective Applications: The paper showcases SCULPTOR's effectiveness in various applications, including:
  • Archaeological Skeletal Facial Completion: Reconstructing faces from partial skeletal remains.
  • Character Fusion: Blending facial features from different individuals.
  • Skull Inference from Image: Estimating skull shape from 2D images.
  • Face Generation with Lipo Level Change: Simulating weight gain/loss effects while maintaining anatomical consistency.
  • Facial Animations: Generating realistic skin deformation under dynamic forces.
  • Compactness and Generalization: Model evaluation showed good compactness (ability to represent data with fewer components) and generalization (performance on unseen data) for shape, trait, and appearance spaces.

Limitations and Future Work

  • Dataset Bias: The LUCY dataset, derived from plastic surgery patients, might have a limited range of skeletal variations, particularly for certain anatomical regions that are not commonly surgically altered. This could limit the expressiveness of the trait component in those areas.
  • Linear Blend Skinning: The model uses Linear Blend Skinning, which can lead to artifacts in extreme deformations.
  • No Muscle Modeling: The current model does not explicitly model facial muscles, relying on skinning and blend shapes.
  • Future Work:
    • Enriching the Dataset: Incorporating data from other sources to increase diversity.
  • Muscle Modeling: Integrating muscle simulation for more realistic facial dynamics.
  • Dynamic MRI Data: Utilizing dynamic MRI data to capture muscle and soft tissue information for more comprehensive modeling.
  • Exploring other skeletal structures: Expanding the model to include more detailed skeletal structures beyond the skull.

Summary updated

Abstract

Recent years have seen growing interest in 3D human face modeling due to its wide applications in digital human, character generation and animation. Existing approaches overwhelmingly emphasized on modeling the exterior shapes, textures and skin properties of faces, ignoring the inherent correlation between inner skeletal structures and appearance. In this paper, we present SCULPTOR, 3D face creations with Skeleton Consistency Using a Learned Parametric facial generaTOR, aiming to facilitate the easy creation of both anatomically correct and visually convincing face models via a hybrid parametric-physical representation. At the core of SCULPTOR is LUCY, the first large-scale shape-skeleton face dataset in collaboration with plastic surgeons. Named after the fossils of one of the oldest known human ancestors, our LUCY dataset contains high-quality Computed Tomography (CT) scans of the complete human head before and after orthognathic surgeries, which are critical for evaluating surgery results. LUCY consists of 144 scans of 72 subjects (31 male and 41 female), where each subject has two CT scans taken pre- and post-orthognathic operations. Based on our LUCY dataset, we learned a novel skeleton consistent parametric facial generator, SCULPTOR, which can create unique and nuanced facial features that help define a character and at the same time maintain physiological soundness. Our SCULPTOR jointly models the skull, face geometry and face appearance under a unified data-driven framework by separating the depiction of a 3D face into shape blend shape, pose blend shape and facial expression blend shape. SCULPTOR preserves both anatomic correctness and visual realism in facial generation tasks compared with existing methods. Finally, we showcase the robustness and effectiveness of SCULPTOR in various fancy applications unseen before, like archaeological skeletal facial completion, bone-aware character fusion, skull inference from images, face generation with lipo-Level change and facial animations, etc.
BibTeX
@article{qiu2022sculptor,
  author = {Zesong Qiu and Yuwei Li and Dongming He and Qixuan Zhang and Longwen Zhang and Yinghao Zhang and Jingya Wang and Lan Xu and Xudong Wang and Yuyao Zhang and Jingyi Yu},
  title = {{SCULPTOR: Skeleton-Consistent Face Creation Using a Learned Parametric Generator}},
  journal = {ACM Trans. Graphics},
  year = {2022},
  doi = {10.1145/3550454.3555462},
}
Figure