Machine Learning Approaches for Prediction of Facial Rejuvenation Using Real and Synthetic Data

Syed Afaq Ali Shah, Mohammed Bennamoun, Michael MoltonIEEE Access 2019 · DOI · OpenReview

Introduction

In the field of cosmetic dermatology, predicting the outcome of a procedure remains a largely subjective endeavor. Practitioners often rely on qualitative descriptions or 2D "before and after" photographs of previous patients to manage expectations. However, these methods fail to account for the unique 3D structural nuances of an individual's face. This paper addresses the critical need for objective, data-driven tools that can quantify and predict how specific volumes of dermal filler—such as 0.5ml versus 1.0ml—will physically alter a patient's facial geometry.

Beyond the clinical challenge, the development of AI in this space is hindered by extreme data scarcity. Unlike general computer vision tasks that benefit from millions of public images, medical aesthetic data is limited by strict privacy laws and ethical considerations. This research introduces a framework to overcome these hurdles using a combination of novel 3D datasets and specialized machine learning models.

Methodology and Data Synthesis

The researchers developed a two-pronged approach to volume prediction, supported by a novel data augmentation technique:

  1. Rejuv3DNet: A custom four-layer deep neural network designed to classify required filler volumes. The model processes 3D facial patches extracted from specific anatomical regions (e.g., cheeks or temples) to predict the appropriate volume class.
  2. Kernel Regression (KR) Model: A statistical model used to validate and refine volume estimation. By modeling the joint distribution of pre-treatment scans and filler volumes, the KR model identifies the "minimum distance" between a new test case and known successful outcomes.
  3. Interactive Mesh Morphing: To address the lack of real-world training data, the authors created an algorithm that modifies 3D facial vertices. This allows for the synthesis of post-treatment faces from standard 3D face databases, effectively "bootstrapping" the training process for deep learning models.
Overview of the facial rejuvenation mapping process: (a) pre-treatment 3D scan, (b) clinical filler distribution plan with volumes, and (c) the resulting post-treatment 3D facial structure.

Datasets

The study utilized three distinct datasets to train and validate the models:

  • Real 3D Face Cosmetic Dataset: A first-of-its-kind collection featuring 80 scans from 40 subjects (pre- and post-treatment) across diverse ethnicities and ages (40–70 years).
  • Synthetic BU-3DEF & GavabDB Datasets: Created by applying the mesh morphing algorithm to existing 3D face databases to provide the volume of data necessary for robust model training.

Key Insights and Results

The transition from 2D to 3D is essential for this application, as 3D scans allow models to learn "structural mapping"—the actual physical displacement of tissue caused by fillers.

Performance Metrics

  • Real-World Accuracy: On the limited real-world dataset, Rejuv3DNet achieved 62.5% accuracy, while the KR model reached 66.67%. These results serve as a proof of concept, demonstrating that even with small datasets, objective prediction is possible.
  • Synthetic Data Success: Performance improved dramatically on larger synthetic sets, with the KR model reaching up to 90.1% accuracy. This suggests that high-quality synthetic data is a viable bridge for medical AI when privacy concerns limit data collection.
  • Clinical Efficiency: The models are highly efficient, with testing times ranging from 0.0005 to 0.0043 seconds. This speed makes the system suitable for real-time use during patient consultations.

Interestingly, the Kernel Regression model often outperformed the deep neural network. This highlights a common reality in "small-data" regimes: when sample sizes are limited, sophisticated statistical regression can be more robust than complex deep learning architectures.

Limitations and Future Outlook

The primary limitation remains the size and balance of the real-world dataset. Current data exhibits a "class imbalance," with more examples of 1ml and 0.5ml treatments than 0.2ml, which occasionally biased the neural network's predictions. Future work will focus on expanding the diversity of real-world data and adapting the mesh morphing technique to simulate more complex surgical outcomes beyond dermal fillers.

Summary updated

Abstract

This paper proposes a novel machine learning approaches to predict the outcome of facial rejuvenation prior to a cosmetic procedure. This is achieved by estimating the required amount of dermal filler volume that needs to be applied on the face by learning the underlying structural mapping from the pretreatment and posttreatment 3D face images. We develop and train our proposed deep neural network, called Rejuv3DNet, designed specifically to predict the dermal filler volume. We also propose the kernel regression (KR)-based model to validate and improve our volume estimation results using regression. Our other contributions include the development of the first 3D face cosmetic dataset, which consists of real-world pretreatment and posttreatment 3D face images and a novel technique for the generation of synthetic cosmetic treatment 3D face images. Our experimental results show that the proposed Rejuv3DNet and the KR model achieve 62.5% and 66.67%, respectively, on real-world data, while these techniques achieve a prediction accuracy of 75.2% and 89.5%, and 77.2% and 90.1% on our two different synthetic datasets. Our proposed techniques have been found to be computationally efficient, achieving near real-time prediction performance. The reported accuracies are our preliminary results for proof of concept, which can be improved with more data. The proposed approach has the potential for further investigation in the cosmetic surgery domain.
BibTeX
@article{shah2019machine,
  author = {Syed Afaq Ali Shah and Mohammed Bennamoun and Michael Molton},
  title = {{Machine Learning Approaches for Prediction of Facial Rejuvenation Using Real and Synthetic Data}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2899379},
}
Figure