TL;DR
This paper introduces a fast and robust facial makeup transfer method that combines layer decomposition with a novel illumination transfer technique. Unlike previous methods that struggle with dark makeup or hair occlusions, this approach achieves realistic results in under two seconds on mobile hardware, making it ideal for real-time virtual "try-on" applications.
What Problem Does This Paper Address?
Facial makeup transfer involves taking the "style" of a reference image and applying it to a user's "source" image. While several methods exist, they face three primary hurdles:
- Dark Makeup: Traditional color-space transfers (like CIELAB) often interpret black or very dark makeup as "no color," resulting in gray, unnatural-looking outputs.
- Illumination Mismatches: If the reference image has shadows or highlights that differ from the source, the transferred makeup often looks like a flat mask or carries over unwanted lighting artifacts.
- Occlusions (Air-Bangs): Many users have hair covering their foreheads ("air-bangs"). Standard algorithms often accidentally transfer makeup onto the hair or fail to blend the skin-hair boundary naturally.
- Computational Cost: Deep learning models (like CycleGANs) provide high quality but are often too slow for real-time use on consumer devices.
Methods
The authors propose a five-step pipeline that avoids the heavy computational load of deep learning in favor of sophisticated image processing:
- Whitening and Smoothing: Pre-processing the source image using OpenCV's Color Balance and Bilateral Filtering to create a "clean canvas."
- Facial Alignment: Using a modified Active Shape Model (ASM) to identify 90 landmark points, followed by an affine transformation to warp the reference image to match the source's geometry.
- Layer Decomposition: Both images are decomposed into three distinct layers using the CIELAB color space:
- Facial Structure Layer: Captures lighting and shading.
- Facial Color Layer: Captures the $a$ and $b$ color channels.
- Facial Detail Layer: Captures fine textures and skin features.
- Layer Transfer:
- Detail: Transferred directly.
- Color: Blended using an alpha-blending approach ($\alpha = 0.95$).
- Structure (The Innovation): Instead of a simple copy, the authors use a specific Illumination Transfer formula. This formula adjusts the source's structure based on the reference's lighting, specifically preventing dark makeup from losing its "depth" and becoming gray.
- Air-Bangs Handling: The system uses a Fully Convolutional Network (FCN) for facial parsing to create a "hard mask" (identifying hair, skin, eyes, etc.). This is converted into a "soft mask" to ensure that makeup is only applied to skin regions, even when partially obscured by hair.
Datasets
The researchers curated two high-resolution datasets for this study:
- Before-makeup dataset: 526 images of faces with nude or very light makeup.
- With-makeup dataset: 878 images featuring a wide variety of makeup styles. These were used to validate the robustness of the transfer across different ethnicities, ages, and styles.
Key Insights and Ideas
- Illumination as a Style Component: The "aha moment" here is treating illumination not just as a condition to be normalized, but as a part of the makeup style itself. By mathematically adjusting the structure layer, they can simulate the "weight" of physical foundation and dark eyeshadow.
- Probability-Based Blending: To solve the "air-bangs" problem, the authors move away from binary masks. By using facial parsing to determine the probability of a pixel being "skin" vs. "hair," they can fuse the original source pixels with the makeup pixels more naturally at the boundaries.
- Efficiency via Traditional CV: The paper demonstrates that for specific tasks like makeup, well-tuned traditional computer vision (layer decomposition + WLS filtering) can outperform deep learning in speed while maintaining competitive quality.
Key Results
- Speed: The proposed method runs in under 2 seconds on an iPhone 6 for a $224 \times 224$ image. In contrast, deep learning methods like Liu et al. (2016) take roughly 6 seconds on a high-end TITAN X GPU.
- Qualitative Superiority: The method successfully transfers black/dark makeup where previous state-of-the-art methods (like Guo et al.) produced gray artifacts.
- User Preference: In a user study on Amazon Mechanical Turk, 87.3% of participants preferred the results of this method over the PairedCycleGAN approach by Chang et al.
Limitations and Future Work
While the paper shows impressive results, a few limitations are inherent:
- Landmark Dependency: The quality of the transfer is heavily dependent on the initial 90-point landmark detection. Extreme poses or heavy occlusions beyond "air-bangs" might cause the alignment to fail.
- Manual Refinement: The authors mention that their software allows for manual landmark adjustment, suggesting that the automatic detection isn't always perfect for all makeup types.
- Future Directions: The authors suggest that expanding the system to handle more complex environmental lighting and even more diverse facial occlusions would be the next logical step.
Summary updated