TL;DR
This paper introduces BeautifAI, a novel makeup recommendation system that personalizes suggestions based on both facial features and the specific occasion. It goes beyond existing methods by offering region-wise recommendations (skin, eyes, lips), real-time makeup previews, and continuous feedback during application, aiming for a more interactive and accurate user experience.
What Problem Does This Paper Address?
The beauty industry is booming, yet current makeup recommendation systems often fall short. They typically offer generic, "one-size-fits-all" recommendations for the entire face and fail to consider the crucial context of the occasion (e.g., casual, professional, party). Furthermore, users lack tools to visualize how a recommended look would appear on their face or receive guidance during the actual application process. This gap leaves users with limited personalization and a less-than-ideal experience, especially in regions like India where access to professional beauty consultants is scarce.
Methods
BeautifAI is built upon four key modules:
- Module 1: Facial Feature Extraction: This module leverages fine-tuned state-of-the-art algorithms (DeepFace for the overall face, Faster R-CNN with ResNet-50 for eyes and lips) to extract detailed facial features. This is crucial for tailoring recommendations to an individual's unique facial structure.
- Module 2: Occasion-oriented Region-wise Makeup Recommendation: This is a core contribution. Instead of a holistic recommendation, it suggests makeup styles for specific regions (skin, eyes, lips) based on the user's chosen occasion. It uses neural embeddings to compare user facial features with a curated dataset, offering a much wider range of personalized options.
- Module 3: Makeup Style Transfer: To provide realistic previews, this module employs a region-wise style transfer technique using Generative Adversarial Networks (GANs). It's trained on a large makeup dataset and fine-tuned to apply makeup styles to specific facial regions, allowing users to visualize the recommended look.
- Module 4: Continuous Makeup Evaluation: This novel module offers real-time feedback to users as they apply makeup. It compares the user's applied makeup (using color correlogram features) against the recommended style, providing guidance on whether to apply more or less makeup to achieve the desired look.
Datasets
The paper introduces a newly curated dataset of 804 annotated images scraped from social media platforms (Instagram and Pinterest). This dataset is unique in that it labels images with both makeup styles and the context of the occasion (Casual, Professional/Office, Party). This differs from existing datasets which often lack occasion-specific labels or comprehensive makeup attribute annotations. The dataset was annotated by three fashion/makeup experts, with a high inter-annotator agreement (kappa values of 0.76 for attributes and 0.88 for occasions).
Key Insights and Ideas
- Occasion is King (for Makeup): The paper strongly emphasizes that makeup recommendations are significantly improved by incorporating the context of the occasion, moving beyond generic suggestions.
- Region-wise is Better: Breaking down makeup recommendations by facial region (skin, eyes, lips) offers far greater personalization and diversity than a single, holistic recommendation.
- Visualizing is Believing: Real-time makeup previews are essential for user engagement and confidence in the recommended styles.
- Guidance During Application Matters: Continuous feedback during makeup application can significantly help users achieve their desired look, bridging the gap between recommendation and execution.
- Neural Embeddings for Nuance: Using neural embeddings for feature comparison allows for finer-grained matching and more accurate recommendations than simple classification.
Key Results
- User Study - Recommendation Diversity & Relevance: Users rated the region-wise recommendations from BeautifAI significantly higher in terms of diversity (0.83 vs. 0.63) and relevance (0.86 vs. 0.71) compared to traditional holistic recommendations.
- User Study - Style Transfer: BeautifAI's style transfer module was rated highly by users for eyes (57%) and lips (45%), though BeautyGAN [12] was preferred for skin (45%).
- User Study - Continuous Evaluation: A majority of users found the continuous evaluation module "Highly Effective" or "Moderately Effective" for all regions (lips: 76.6%, eyes: 86.6%, skin: 83.2%).
- Color Correlogram Efficacy: The color correlogram algorithm was found to be the most accurate for continuous makeup evaluation across all regions (eyes: 68%, lips: 55%, skin: 53%), outperforming color histograms and SURF.
Limitations and Future Work
- Dataset Demographics: The current dataset and user study primarily focus on females aged 18-45. Future work aims to expand the dataset to cover a broader demographic.
- Advanced Personalization: The authors plan to incorporate higher-level makeup concepts such as ethnicity, cultural dependence, time of day, body features, and dress to further personalize recommendations.
- Broader Occasion Coverage: While three occasions are covered, expanding to more nuanced or specific events could enhance the system.
- Real-world Deployment: The paper focuses on the technical aspects; future work might explore seamless integration into mobile applications or other user-facing platforms.
Summary updated