TL;DR
This paper introduces SARA, a novel framework for controllable makeup transfer that effectively handles significant pose misalignments between source and reference images. By integrating semantic-guided alignment with unbalanced optimal transport and region-adaptive normalization, SARA achieves fine-grained control over makeup application, including partial transfer and intensity adjustment, while preserving facial identity.
What Problem Does This Paper Address?
Applying makeup virtually to a target face image while preserving the original identity is a challenging task, especially when the source and reference images have different poses or facial orientations. Existing methods often struggle with these misalignments, leading to unnatural results or an inability to transfer specific makeup styles accurately. Furthermore, achieving fine-grained control over the makeup transfer, such as applying only certain makeup elements (e.g., lipstick, eyeshadow) or adjusting the intensity of the makeup, is crucial for real-world applications but is often lacking in current approaches. This paper aims to address these limitations by proposing a framework that can handle significant pose variations and offers flexible control over the makeup transfer process.
Methods
SARA is built upon three core modules:
- Semantic-guided Alignment Module (SAM): This module tackles pose misalignment by explicitly constructing dense correspondences between the reference and source images. It leverages unbalanced optimal transport to match semantic regions, which is more robust than traditional methods that rely on landmarks or cosine similarity, especially when dealing with unequal distributions of features. This ensures that makeup styles are warped accurately even with significant pose differences.
- Region-Adaptive Normalization Module (RAM): To compensate for potential feature loss during alignment and to enable shape-independent style control, RAM extracts region-specific style codes from the reference image. These codes are then broadcast to the corresponding regions of the target semantic map. This module dynamically combines these shape-independent style codes with the warped output from SAM, enhancing the detail and sharpness of the transferred makeup.
- Makeup Fusion Module (MFM): This module progressively fuses the identity features of the source image with the modulated makeup styles to generate the final, realistic output image. It uses a face identity encoder to preserve the source identity and then integrates the processed makeup styles.
Datasets
The authors trained their model using the Makeup Transfer (MT) dataset, which contains 2719 makeup images and 1115 non-makeup images. For testing, they randomly selected 250 makeup and 100 non-makeup images from the MT dataset. They also evaluated their model on the M-wild dataset to assess performance on facial images with complex backgrounds and significant pose variations.
Key Insights and Ideas
- Unbalanced Optimal Transport for Alignment: The paper highlights the effectiveness of unbalanced optimal transport in establishing dense correspondences for feature alignment, particularly when dealing with misaligned poses and semantic region mismatches. This is a key innovation for robust makeup transfer.
- Region-Adaptive Normalization for Detail Preservation: RAM's approach of using shape-independent style codes and dynamically combining them with warped features is crucial for maintaining fine details and compensating for information loss during alignment, leading to more natural results.
- Reversibility of Makeup Transfer: The framework also supports makeup removal by treating the non-makeup image as the reference and the makeup image as the source, demonstrating the process's reversibility.
- Controllable Transfer: The ability to perform partial makeup transfer (e.g., only lipstick) and shade-controllable transfer (adjusting makeup intensity) through linear interpolation offers significant flexibility for user-driven applications.
Key Results
- Qualitative Results: SARA demonstrated superior performance compared to existing methods like PSGAN, SCGAN, EleGANt, and SSAT, especially in handling large spatial misalignments and occlusions. The results showed more accurate transfer of eye shadow, lipstick, and blush, with fewer artifacts and more natural skin tones.
- Quantitative Results: User studies indicated that SARA achieved the highest human preference scores for both transfer accuracy and image quality, outperforming other methods in both aligned and misaligned scenarios.
Limitations and Future Work
- Limitations: The paper acknowledges that while their method handles large geometric changes better than TPS warping, there might still be limitations in preserving extremely intricate makeup details under severe misalignments. The effectiveness of pseudo ground truth generation, while improved, is still an area where further refinement could be beneficial.
- Future Work: The authors suggest that further research could explore more advanced techniques for generating pseudo ground truth and potentially investigate more sophisticated methods for handling extreme pose variations or complex facial expressions. The framework's ability to be reversed for makeup removal also opens avenues for further exploration in related image editing tasks.
Summary updated