Facial Makeup Transfer Combining Illumination Transfer

Xin Jin, Rui Han, Ning Ning, Xiaodong Li, Xiaokun Zhang

TL;DR

This paper introduces a fast and robust facial makeup transfer method that combines layer decomposition with a novel illumination transfer technique. Unlike previous methods that struggle with dark makeup or hair occlusions, this approach achieves realistic results in under two seconds on mobile hardware, making it ideal for real-time virtual "try-on" applications.

What Problem Does This Paper Address?

Facial makeup transfer involves taking the "style" of a reference image and applying it to a user's "source" image. While several methods exist, they face three primary hurdles:

  1. Dark Makeup: Traditional color-space transfers (like CIELAB) often interpret black or very dark makeup as "no color," resulting in gray, unnatural-looking outputs.
  2. Illumination Mismatches: If the reference image has shadows or highlights that differ from the source, the transferred makeup often looks like a flat mask or carries over unwanted lighting artifacts.
  3. Occlusions (Air-Bangs): Many users have hair covering their foreheads ("air-bangs"). Standard algorithms often accidentally transfer makeup onto the hair or fail to blend the skin-hair boundary naturally.
  4. Computational Cost: Deep learning models (like CycleGANs) provide high quality but are often too slow for real-time use on consumer devices.

Methods

The authors propose a five-step pipeline that avoids the heavy computational load of deep learning in favor of sophisticated image processing:

  1. Whitening and Smoothing: Pre-processing the source image using OpenCV's Color Balance and Bilateral Filtering to create a "clean canvas."
  2. Facial Alignment: Using a modified Active Shape Model (ASM) to identify 90 landmark points, followed by an affine transformation to warp the reference image to match the source's geometry.
  3. Layer Decomposition: Both images are decomposed into three distinct layers using the CIELAB color space:
  4. Facial Structure Layer: Captures lighting and shading.
  5. Facial Color Layer: Captures the $a$ and $b$ color channels.
  6. Facial Detail Layer: Captures fine textures and skin features.
  7. Layer Transfer:
    • Detail: Transferred directly.
  8. Color: Blended using an alpha-blending approach ($\alpha = 0.95$).
  9. Structure (The Innovation): Instead of a simple copy, the authors use a specific Illumination Transfer formula. This formula adjusts the source's structure based on the reference's lighting, specifically preventing dark makeup from losing its "depth" and becoming gray.
  10. Air-Bangs Handling: The system uses a Fully Convolutional Network (FCN) for facial parsing to create a "hard mask" (identifying hair, skin, eyes, etc.). This is converted into a "soft mask" to ensure that makeup is only applied to skin regions, even when partially obscured by hair.

Datasets

The researchers curated two high-resolution datasets for this study:

  • Before-makeup dataset: 526 images of faces with nude or very light makeup.
  • With-makeup dataset: 878 images featuring a wide variety of makeup styles. These were used to validate the robustness of the transfer across different ethnicities, ages, and styles.

Key Insights and Ideas

  • Illumination as a Style Component: The "aha moment" here is treating illumination not just as a condition to be normalized, but as a part of the makeup style itself. By mathematically adjusting the structure layer, they can simulate the "weight" of physical foundation and dark eyeshadow.
  • Probability-Based Blending: To solve the "air-bangs" problem, the authors move away from binary masks. By using facial parsing to determine the probability of a pixel being "skin" vs. "hair," they can fuse the original source pixels with the makeup pixels more naturally at the boundaries.
  • Efficiency via Traditional CV: The paper demonstrates that for specific tasks like makeup, well-tuned traditional computer vision (layer decomposition + WLS filtering) can outperform deep learning in speed while maintaining competitive quality.

Key Results

  • Speed: The proposed method runs in under 2 seconds on an iPhone 6 for a $224 \times 224$ image. In contrast, deep learning methods like Liu et al. (2016) take roughly 6 seconds on a high-end TITAN X GPU.
  • Qualitative Superiority: The method successfully transfers black/dark makeup where previous state-of-the-art methods (like Guo et al.) produced gray artifacts.
  • User Preference: In a user study on Amazon Mechanical Turk, 87.3% of participants preferred the results of this method over the PairedCycleGAN approach by Chang et al.

Limitations and Future Work

While the paper shows impressive results, a few limitations are inherent:

  • Landmark Dependency: The quality of the transfer is heavily dependent on the initial 90-point landmark detection. Extreme poses or heavy occlusions beyond "air-bangs" might cause the alignment to fail.
  • Manual Refinement: The authors mention that their software allows for manual landmark adjustment, suggesting that the automatic detection isn't always perfect for all makeup types.
  • Future Directions: The authors suggest that expanding the system to handle more complex environmental lighting and even more diverse facial occlusions would be the next logical step.

Summary updated

Abstract

To meet the women appearance needs, we present a novel virtual experience approach of facial makeup transfer, developed into windows platform application software. The makeup effects could present on the user's input image in real time, with an only single reference image. The input image and reference image are divided into three layers by facial feature points landmarked: facial structure layer, facial color layer, and facial detail layer. Except for the above layers are processed by different algorithms to generate output image, we also add illumination transfer, so that the illumination effect of the reference image is automatically transferred to the input image. Our approach has the following three advantages: (1) Black or dark and white facial makeup could be effectively transferred by introducing illumination transfer; (2) Efficiently transfer facial makeup within seconds compared to those methods based on deep learning frameworks; (3) Reference images with the air-bangs could transfer makeup perfectly.
BibTeX
@article{jin2019facial,
  author = {Xin Jin and Rui Han and Ning Ning and Xiaodong Li and Xiaokun Zhang},
  title = {{Facial Makeup Transfer Combining Illumination Transfer}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2923116},
}

Loading related content...

Figure