Evaluating 3D human face reconstruction from a frontal 2D image, focusing on facial regions associated with foetal alcohol syndrome

Felix Atuhaire, Bernhard Egger, Tinashe MutsvangwaSouth 2023 · DOI

TL;DR

Researchers have developed a framework to reconstruct accurate 3D facial models from a single 2D smartphone-style photo to aid in screening for Foetal Alcohol Syndrome (FAS). By using 3D Morphable Models (3DMMs), the team achieved a reconstruction accuracy of 2.62 mm, offering a potentially low-cost, accessible alternative to expensive 3D clinical scanners.

What Problem Does This Paper Address?

Foetal Alcohol Syndrome (FAS) is a preventable but permanent condition caused by prenatal alcohol exposure. Early diagnosis is critical for intervention, but it relies heavily on identifying a specific "facial phenotype"—features like a smooth philtrum (the groove above the upper lip), a thin upper lip, and short eye openings (palpebral fissures).

Currently, accurate measurement requires either expert dysmorphologists or expensive 3D surface scanners, both of which are scarce in low-resource settings like parts of South Africa, where FAS prevalence is high. While 2D photos are easy to take, they lack the geometric depth needed for precise clinical measurement. This paper investigates whether "inverse graphics"—reconstructing a 3D face from a 2D image—is accurate enough to be used for FAS screening.

Methods

The authors employed 3D Morphable Models (3DMMs), which are generative models that capture how human faces vary in shape and texture.

  1. Model Building: They used a database of 3D scans to create a "mean face" and used Principal Component Analysis (PCA) to define the primary ways faces deviate from that mean.
  2. Registration: They used a Gaussian process fitting approach to ensure "dense correspondence"—meaning every point on one face (like the tip of the nose) is mathematically mapped to the same point on every other face in the dataset.
  3. Reconstruction: To turn a 2D image into 3D, they used a model-driven sampling algorithm. This algorithm essentially "guesses" the 3D shape and texture parameters that, when rendered by a virtual camera, would most closely match the input 2D photo.
  4. Evaluation: They used a leave-one-out cross-validation strategy, measuring the distance between the reconstructed 3D surface and the actual "ground truth" 3D scan.

Datasets

The study utilized the BU-3DFE (Binghamton University 3D Facial Expression) database.

  • Composition: 98 subjects (56 female, 42 male) aged 18–70.
  • Diversity: Unlike many previous benchmarks that are biased toward lighter skin tones, this dataset includes diverse ethnicities, which is crucial for a tool intended for global health applications.

Key Insights and Ideas

  • Skin Tone Neutrality: One of the most significant findings was that reconstruction quality was not significantly affected by skin tone. Previous 3D scanning technologies often struggled with darker skin; this model-based approach proved robust across the spectrum.
  • Region-Specific Accuracy: The errors were not uniform. Interestingly, the regions most critical for FAS diagnosis—the midface and the area around the eyes—showed lower levels of surface variation and higher reconstruction accuracy than "outlier" regions like the outer cheeks or chin.
  • The "Inverse Graphics" Advantage: By fitting a 3D model rather than just measuring pixels in 2D, the system can estimate 3D landmark positions even if the face is slightly turned, potentially making it more "forgiving" than traditional 2D photogrammetry.

Key Results

  • Overall Accuracy: The average Root Mean Square (RMS) error for the entire face surface was 2.62 mm.
  • Landmark Precision: For 14 critical facial landmarks, 11 had an error of less than 3.5 mm.
  • FAS Metrics: The error for palpebral fissure length (eye width) was only 1.25 mm, which is highly promising for clinical utility.
  • Model Compactness: The team found that just 20 principal components could account for over 90% of the facial shape variation in their dataset.

Limitations and Future Work

The authors acknowledge several hurdles before this hits the clinic:

  • Subject Demographic: The model was trained and tested on normal adult controls, not children or individuals with confirmed FAS. Children's faces have different proportions, and the specific anomalies of FAS might require a model trained on that specific population.
  • Input Data: The 2D images used were rendered from 3D scans in a controlled environment. "In-the-wild" photos from various smartphones with different lighting and focal lengths will introduce more noise.
  • Automation: While the framework is largely automated, it still used 12 manual landmarks to initialize the process. Future versions aim for a completely "one-click" reconstruction.

The ultimate goal is to integrate this into a mobile app, allowing healthcare workers in remote areas to screen for FAS using nothing more than a standard smartphone camera.

Summary updated

Abstract

Foetal alcohol syndrome (FAS) is a preventable condition caused by maternal alcohol consumption during pregnancy. The FAS facial phenotype is an important factor for diagnosis, alongside central nervous system impairments and growth abnormalities. Current methods for analysing the FAS facial phenotype rely on 3D facial image data, obtained from costly and complex surface scanning devices. An alternative is to use 2D images, which are easy to acquire with a digital camera or smart phone. However, 2D images lack the geometric accuracy required for accurate facial shape analysis. Our research offers a solution through the reconstruction of 3D human faces from single or multiple 2D images. We have developed a framework for evaluating 3D human face reconstruction from a single-input 2D image using a 3D face model for potential use in FAS assessment. We first built a generative morphable model of the face from a database of registered 3D face scans with diverse skin tones. Then we applied this model to reconstruct 3D face surfaces from single frontal images using a model-driven sampling algorithm. The accuracy of the predicted 3D face shapes was evaluated in terms of surface reconstruction error and the accuracy of FAS-relevant landmark locations and distances. Results show an average root mean square error of 2.62 mm. Our framework has the potential to estimate 3D landmark positions for parts of the face associated with the FAS facial phenotype. Future work aims to improve on the accuracy and adapt the approach for use in clinical settings. Significance: Our study presents a framework for constructing and evaluating a 3D face model from 2D face scans and evaluating the accuracy of 3D face shape predictions from single images. The results indicate low generalisation error and comparability to other studies. The reconstructions also provide insight into specific regions of the face relevant to FAS diagnosis. The proposed approach presents a potential cost-effective and easily accessible imaging tool for FAS screening, yet its clinical application needs further research.
BibTeX
@inproceedings{atuhaire2023evaluating,
  author = {Felix Atuhaire and Bernhard Egger and Tinashe Mutsvangwa},
  title = {{Evaluating 3D human face reconstruction from a frontal 2D image, focusing on facial regions associated with foetal alcohol syndrome}},
  booktitle = {South},
  year = {2023},
  doi = {10.17159/sajs.2023/12064},
}
Figure