TL;DR
This paper introduces SCULPTOR, a novel system for generating realistic 3D human faces that explicitly models the underlying skeletal structure (skull) alongside facial geometry and appearance. By leveraging a new large-scale dataset of CT scans and 3D face scans, SCULPTOR enables more anatomically consistent and characteristic facial variations than previous methods, opening doors for applications like forensic reconstruction and character design.
What Problem Does This Paper Address?
Creating realistic and diverse 3D human faces is crucial for applications ranging from digital avatars to medical simulations. While existing methods have made significant strides in modeling external facial features, textures, and expressions, they often overlook the fundamental anatomical relationship between the skull and the face. This oversight can lead to anatomically implausible or unconvincing facial variations. The paper argues that facial appearance is biomechanically linked to skeletal structures, and that a consistent facial model should respect these underlying anatomical rules. The challenge is to develop a system that can generate faces with high anatomical correctness and visual realism, allowing for nuanced control over facial features that are rooted in skeletal variations.
Methods
SCULPTOR employs a hybrid parametric-physical representation to model 3D faces. The core of the system is a learned parametric generator that jointly models the skull, face geometry, and face appearance. This is achieved by:
- Skeleton-Consistent Parametric Model: SCULPTOR extends existing parametric face models (like FLAME) by explicitly incorporating the skull's influence. It models face geometry as a combination of shape, trait, pose, and expression blend shapes, where the "trait" component specifically captures variations influenced by skeletal structure.
- Linear Blend Skinning (LBS): The model uses LBS to deform the face mesh based on skeletal poses and learned skinning weights.
- Trait Component: A novel parameter, the "trait component" ($\gamma$), is introduced to represent physiologically sound variations in the internal skeleton and their influence on face shape. This is learned from pre- and post-surgery CT data, capturing how skeletal changes affect external appearance.
- Data-Driven Framework: The entire model is trained in a data-driven manner using a comprehensive dataset.
- Registration and Parameter Learning: A multi-stage process involves registering template skull and face meshes to the dataset, followed by iterative learning of shape, trait, pose, and appearance parameters.
Datasets
- LUCY Dataset: The paper introduces LUCY, a new large-scale dataset specifically designed for this research. It comprises high-quality Computed Tomography (CT) scans of complete human heads from 72 subjects (31 male, 41 female), with scans taken both before and after orthognathic surgeries. This dataset is crucial as it provides paired pre- and post-surgery data, allowing for the study of how skeletal changes influence facial appearance. It also includes 3D facial scans with texture maps.
- FaceScape Dataset: Used for learning pose-related parameters (skinning weight and pose blend shapes).
- 3DSCANSTORE: Used for additional face texture assets.
- FFHQ Dataset: Used for training the skull inference model.
Key Insights and Ideas
- Skull as the Foundation: The paper strongly emphasizes that the skull is the primary determinant of facial structure and appearance. Ignoring this underlying anatomy leads to less convincing and anatomically incorrect facial variations.
- Trait Space for Nuance: The introduction of a "trait space" is a key innovation. This allows for controlling subtle, anatomically-grounded variations in facial features (like jaw width or length) that are directly linked to skeletal changes, going beyond global shape adjustments.
- Bridging Medical Imaging and Graphics: The use of CT scans from plastic surgery provides a unique bridge between medical imaging data and computer graphics, enabling the learning of anatomically informed facial models.
- Disentangling Factors: The model aims to disentangle shape, trait, pose, and expression, allowing for more controlled and interpretable editing of facial characteristics.
Key Results
- Improved Skull Fitting: SCULPTOR-SIMPLE (without trait components) and SCULPTOR (with trait components) both showed improved skull fitting performance (lower Mean Vertex Squared Error) compared to baseline methods on test data. SCULPTOR with trait components generally outperformed SCULPTOR-SIMPLE.
- Realistic Facial Variations: Qualitative results demonstrate SCULPTOR's ability to generate diverse and characteristic facial variations, including those influenced by skeletal changes (e.g., jaw shape, cheekbone height).
- Effective Applications: The paper showcases SCULPTOR's effectiveness in various applications, including:
- Archaeological Skeletal Facial Completion: Reconstructing faces from partial skeletal remains.
- Character Fusion: Blending facial features from different individuals.
- Skull Inference from Image: Estimating skull shape from 2D images.
- Face Generation with Lipo Level Change: Simulating weight gain/loss effects while maintaining anatomical consistency.
- Facial Animations: Generating realistic skin deformation under dynamic forces.
- Compactness and Generalization: Model evaluation showed good compactness (ability to represent data with fewer components) and generalization (performance on unseen data) for shape, trait, and appearance spaces.
Limitations and Future Work
- Dataset Bias: The LUCY dataset, derived from plastic surgery patients, might have a limited range of skeletal variations, particularly for certain anatomical regions that are not commonly surgically altered. This could limit the expressiveness of the trait component in those areas.
- Linear Blend Skinning: The model uses Linear Blend Skinning, which can lead to artifacts in extreme deformations.
- No Muscle Modeling: The current model does not explicitly model facial muscles, relying on skinning and blend shapes.
- Future Work:
- Enriching the Dataset: Incorporating data from other sources to increase diversity.
- Muscle Modeling: Integrating muscle simulation for more realistic facial dynamics.
- Dynamic MRI Data: Utilizing dynamic MRI data to capture muscle and soft tissue information for more comprehensive modeling.
- Exploring other skeletal structures: Expanding the model to include more detailed skeletal structures beyond the skull.
Summary updated