A smartphone application for personalized facial aesthetic monitoring

Wataru Hashimoto, Shohei KanedaSkin 2024 · DOI

TL;DR

This paper introduces a smartphone app that uses AI-powered facial landmark detection to ensure consistent facial image capture for personalized aesthetic monitoring. The app aims to provide a more accessible and affordable alternative to professional systems for tracking skin quality changes and evaluating skincare treatments at home.

What Problem Does This Paper Address?

Assessing and monitoring facial skin quality and the effectiveness of skincare treatments at home is challenging. Existing professional-grade systems are expensive and impractical for everyday users. While smartphones offer ubiquity, capturing consistent facial images over time is difficult due to variations in lighting, face position, and orientation. This inconsistency hinders accurate tracking of subtle skin changes, making it hard for individuals to objectively evaluate their skincare routines.

Methods

The core innovation is the Face Alignment Indicators (FAIN) system, integrated into an iOS camera application. This system utilizes Facial Landmark Detection (FLD), an AI technique, to identify key facial features. The FAIN system employs a fixed target indicator and a dynamic alignment indicator that guides the user to position their face consistently. When the user's face aligns with the indicator, the app automatically captures the image.

Key aspects of the method include:

  • AI-driven Face Alignment: FLD provides 74 landmark points, used to create a triangle and line indicator. Users align their face to match this indicator with a fixed target.
  • Standardized Capture Conditions: The app prompts users to measure ambient light (aiming for 0 lx) and maximize screen brightness, which also acts as an illumination source.
  • Geometric and Colorimetric Analysis: The study evaluated consistency using geometric data (face size, position, pitch, and yaw) and colorimetric data from applied stickers.
  • Personalized Target Indicators: For each user, a "criterion image" is used to generate personalized target indicators, improving alignment accuracy.

Datasets

The study did not use pre-existing datasets for training. Instead, it involved:

  • Mannequin Head: Used for initial in vitro testing of the FAIN system.
  • Eight Participants: Their facial images were captured under various conditions (fixed vs. handheld smartphone, tolerant vs. strict alignment criteria) for evaluating app usability and face appearance uniformity.
  • Skin-colored Stickers: Used in an in vivo feasibility study to simulate pseudo-skin color changes on a participant's cheek.

Key Insights and Ideas

  • Democratizing Aesthetic Monitoring: The app offers a pathway for consumers to perform objective, albeit relative, self-monitoring of their skin, bridging the gap between expensive professional tools and subjective self-assessment.
  • The Power of Consistency: The FAIN system's success in ensuring consistent facial appearance is crucial. Even with variations in face position (distance D) and size (normalized area A), color measurement variability (CV values) remained stable, suggesting the system is robust.
  • Tolerant Criteria Suffice for Color Measurement: While strict alignment criteria improve face position consistency, the study found that tolerant criteria (k=0.05) are sufficient for reliable color measurements, enhancing usability without sacrificing data quality.
  • Subtle Color Changes are Detectable: The in vivo study demonstrated that the app can detect small color differences (∆Eab* of 1.901), which are considered experimentally validated detection limits, indicating potential for tracking subtle skin changes like pigmentation or tanning.

Key Results

  • Usability: The success rate for capturing images within 120 seconds was high, with no significant difference under tolerant criteria. Capture time per image was significantly shorter when the smartphone was fixed.
  • Facial Appearance Uniformity:
    • Distances between target and acquired face positions were significantly shorter when the smartphone was fixed.
  • Normalized area (face size) showed no significant difference between fixed and handheld conditions under strict criteria.
  • Color measurement variability (CV values) for L, a, and b* showed no significant differences across the four conditions, indicating consistent color capture.
  • Color Difference Detection: The smallest color difference (∆Eab*) observed was 1.901, considered a validated detection limit for the app.

Limitations and Future Work

  • Relative Quantification: The method provides relative, not absolute, quantification of skin changes due to variations in camera-subject distance and individual face geometry.
  • Smartphone Model Dependency: Comparisons between images captured with different smartphone models are not feasible due to variations in camera and screen performance.
  • External Software Dependency: Color measurements currently require external software like Photoshop and MATLAB.
  • Frontal Face Specialization: The FAIN system is optimized for frontal faces, requiring different indicators for side-facing views.
  • Mouth Shape Inconsistency: The current FAIN system lacks specific criteria for mouth shape, which can lead to color measurement inconsistencies.
  • Future Work: The authors suggest tightening criteria for the triangular area of the alignment indicator to improve distance consistency and color measurement precision. They also envision broader applications beyond skincare, including monitoring treatment responses for skin diseases and detecting signs of cognitive decline.

Summary updated

Abstract

While the current monitoring method is a relative quantification approach, it contributes to evidence-based evaluations of skincare treatments.
BibTeX
@inproceedings{hashimoto2024smartphone,
  author = {Wataru Hashimoto and Shohei Kaneda},
  title = {{A smartphone application for personalized facial aesthetic monitoring}},
  booktitle = {Skin},
  year = {2024},
  doi = {10.1111/srt.13824},
}
Figure