TL;DR
This paper introduces LiveBeauty, the first large-scale, open-source dataset for facial attractiveness prediction (FAP) specifically tailored for live streaming environments. They also propose FPEM, a novel multi-modal method that leverages facial prior knowledge and aesthetic semantic features to achieve state-of-the-art performance in this challenging domain.
What Problem Does This Paper Address?
Predicting facial attractiveness (FAP) is a valuable task with applications ranging from content recommendation to virtual try-ons. However, existing FAP datasets are often small, proprietary, or lack diversity, making them unsuitable for real-world scenarios like live streaming. Live streaming presents unique challenges due to unpredictable lighting, expressions, and camera angles, which existing models struggle to generalize to. Furthermore, current FAP models often rely on simplistic feature fusion, failing to capture the nuanced interplay of individual facial features and contextual information, thus limiting their adaptability.
Methods
The authors propose a multi-modal method called Facial Prior Enhanced Multi-modal (FPEM), which consists of four main modules:
- Personalized Attractiveness Prior Module (PAPM): This module extracts holistic facial prior knowledge by combining multi-scale visual features from a Swin Transformer with face-aware features from a pre-trained FaceNet. A cross-attention mechanism fuses these features to create a personalized attractiveness representation.
- Multi-modal Attractiveness Encoder Module (MAEM): This module leverages CLIP to extract multi-modal aesthetic semantic features. It uses a text encoder for predefined attractiveness descriptions (e.g., "a photo of a person with X attractiveness") and an image encoder for the input image. Cosine similarity is used to predict attractiveness probabilities.
- Cross-Modal Fusion Module (CMFM): This module refines the textual embeddings from MAEM using the personalized attractiveness features from PAPM, generating personalized textual embeddings for similarity regression.
- Decision Fusion Module (DFM): This module combines the predictions from the previous modules (Q1, Q2, and Q3) through learnable linear layers to produce the final attractiveness score.
The training involves a two-stage strategy: preliminary training of PAPM and MAEM, followed by hybrid fusion with CMFM and DFM.
Datasets
- LiveBeauty (New Benchmark): The paper introduces LiveBeauty, a new, large-scale, open-source FAP dataset specifically for live streaming. It contains 10,000 face images collected from a popular live streaming platform, with 200,000 attractiveness annotations from 20 annotators. The dataset is designed to be more challenging due to the diverse and unpredictable conditions of live streams.
- SCUT-FBP5500 [5]: An existing open-source FAP dataset used for evaluation.
- MEBeauty [6]: Another existing open-source FAP dataset used for evaluation.
Key Insights and Ideas
- The critical need for live-streaming specific FAP data: Existing datasets are insufficient for the dynamic and challenging nature of live streaming, highlighting a significant gap in research.
- Personalized facial priors are crucial: Incorporating individual facial prior knowledge (beyond generic features) significantly improves FAP performance, as demonstrated by the PAPM module.
- Multi-modal fusion enhances robustness: Combining visual and textual information through modules like MAEM and CMFM leads to more comprehensive and accurate attractiveness predictions.
- A staged approach to fusion is effective: The proposed FPEM method's two-stage training and fusion strategy allows for effective integration of different feature types and refinement of predictions.
Key Results
- The proposed FPEM method achieves state-of-the-art performance on the new LiveBeauty dataset, as well as on the existing SCUT-FBP5500 and MEBeauty datasets.
- On LiveBeauty, FPEM achieves an SROCC of 0.925, PLCC of 0.892, and KROCC of 0.773, outperforming other methods by a notable margin.
- The ablation studies confirm the effectiveness of each proposed module (PAPM, MAEM, CMFM) and the overall FPEM architecture.
- Cross-dataset experiments show that FPEM exhibits better generalization capabilities compared to other methods, even when trained on LiveBeauty and tested on datasets with different ethnic compositions.
Limitations and Future Work
- Domain Gap: While FPEM shows improved generalization, a performance drop is observed when tested on datasets with different ethnic compositions and from non-live streaming scenarios, indicating the continued challenge of domain adaptation.
- Future Work: The authors suggest that their LiveBeauty dataset and FPEM method can be effectively leveraged for facial attractiveness prediction in other video applications. Further research could explore more advanced cross-modal fusion techniques and investigate the impact of different demographic factors on attractiveness perception.
Summary updated