Facial Makeup Transfer Combining Illumination Transfer

Xin Jin, Rui Han, Ning Ning, Xiaodong Li, Xiaokun Zhang

Abstract

To meet the growing demand for virtual beauty experiences, we present a novel facial makeup transfer approach developed for real-time applications. Our method allows makeup effects from a single reference image to be applied to a user's input image instantaneously. By decomposing images into facial structure, color, and detail layers and introducing a specialized illumination transfer algorithm, our system overcomes traditional hurdles such as dark makeup rendering and lighting mismatches. The approach is highly efficient, processing transfers in under two seconds on mobile hardware, and effectively handles complex occlusions like "air-bangs."

Introduction

Facial makeup transfer involves migrating the cosmetic style of a reference image onto a source image while preserving the source's identity. Despite its popularity in virtual "try-on" apps, existing methods often struggle with several key issues:

  1. Dark Makeup Degradation: Traditional color-space transfers (e.g., CIELAB) frequently fail to render black or dark makeup accurately, often producing flat, gray artifacts.
  2. Illumination Inconsistency: Differences in lighting between the source and reference images can lead to unnatural results where the makeup looks like a floating mask.
  3. Facial Occlusions: Hair covering the forehead (air-bangs) often interferes with landmark detection and blending, leading to makeup being incorrectly applied to the hair.
  4. Computational Latency: Many high-quality deep learning models, such as CycleGANs, are too computationally expensive for real-time use on consumer mobile devices.

Proposed Methodology

Our pipeline avoids the heavy overhead of deep learning by utilizing a sophisticated image processing framework consisting of five primary stages:

1. Pre-processing

The source image undergoes whitening and smoothing using OpenCV’s Color Balance and Bilateral Filtering. This creates a "clean canvas" by reducing skin irregularities before the makeup is applied.

2. Facial Alignment

We utilize a modified Active Shape Model (ASM) to identify 90 facial landmark points. An affine transformation then warps the reference image to match the geometric structure of the source face.

3. Layer Decomposition and Transfer

Both images are decomposed into three layers within the CIELAB color space:

  • Facial Color Layer: Captures the $a$ and $b$ color channels, which are blended using an alpha-blending approach ($\alpha = 0.95$).
  • Facial Detail Layer: Captures fine skin textures and is transferred directly to maintain realism.
  • Facial Structure Layer: This layer captures lighting and shading. Our core innovation is a specific Illumination Transfer formula that adjusts the source's structure based on the reference's lighting. This prevents dark makeup from losing depth and becoming gray.
A typical reference image featuring heavy green eyeshadow used to test the transfer algorithm.

4. Handling Occlusions (Air-Bangs)

To handle hair occlusions, we employ a Fully Convolutional Network (FCN) for facial parsing. This generates a probability-based "soft mask" that distinguishes between skin and hair, ensuring makeup is only applied to skin regions even when partially obscured by hair.

Experimental Results

The robustness of our method was validated using two high-resolution datasets: a "before-makeup" set of 526 images and a "with-makeup" set of 878 images.

Performance and Speed

Our method demonstrates significant efficiency gains over deep learning frameworks. While models like Liu et al. (2016) require approximately 6 seconds on a high-end TITAN X GPU, our algorithm processes a $224 \times 224$ image in under 2 seconds on an iPhone 6.

Qualitative and User Evaluation

By treating illumination as a component of the makeup style, our method successfully transfers dark eyeshadows and foundations that previous state-of-the-art methods (like Guo et al.) failed to render correctly. In a user study conducted via Amazon Mechanical Turk, 87.3% of participants preferred our results over the PairedCycleGAN approach.

The final transfer result, demonstrating how the green eyeshadow style is realistically blended onto the source face while maintaining natural lighting.

Conclusion

This paper introduces a fast, robust, and realistic solution for facial makeup transfer. By combining traditional layer decomposition with a novel illumination transfer technique, we provide a method that handles dark makeup and hair occlusions more effectively than previous approaches. While the system is dependent on accurate landmark detection, its speed and quality make it a superior choice for real-time virtual beauty applications.

Summary updated

Abstract

To meet the women appearance needs, we present a novel virtual experience approach of facial makeup transfer, developed into windows platform application software. The makeup effects could present on the user's input image in real time, with an only single reference image. The input image and reference image are divided into three layers by facial feature points landmarked: facial structure layer, facial color layer, and facial detail layer. Except for the above layers are processed by different algorithms to generate output image, we also add illumination transfer, so that the illumination effect of the reference image is automatically transferred to the input image. Our approach has the following three advantages: (1) Black or dark and white facial makeup could be effectively transferred by introducing illumination transfer; (2) Efficiently transfer facial makeup within seconds compared to those methods based on deep learning frameworks; (3) Reference images with the air-bangs could transfer makeup perfectly.
BibTeX
@article{jin2019facial,
  author = {Xin Jin and Rui Han and Ning Ning and Xiaodong Li and Xiaokun Zhang},
  title = {{Facial Makeup Transfer Combining Illumination Transfer}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2923116},
}
Figure