Abstract
To meet the growing demand for virtual beauty experiences, we present a novel facial makeup transfer approach developed for real-time applications. Our method allows makeup effects from a single reference image to be applied to a user's input image instantaneously. By decomposing images into facial structure, color, and detail layers and introducing a specialized illumination transfer algorithm, our system overcomes traditional hurdles such as dark makeup rendering and lighting mismatches. The approach is highly efficient, processing transfers in under two seconds on mobile hardware, and effectively handles complex occlusions like "air-bangs."
Introduction
Facial makeup transfer involves migrating the cosmetic style of a reference image onto a source image while preserving the source's identity. Despite its popularity in virtual "try-on" apps, existing methods often struggle with several key issues:
- Dark Makeup Degradation: Traditional color-space transfers (e.g., CIELAB) frequently fail to render black or dark makeup accurately, often producing flat, gray artifacts.
- Illumination Inconsistency: Differences in lighting between the source and reference images can lead to unnatural results where the makeup looks like a floating mask.
- Facial Occlusions: Hair covering the forehead (air-bangs) often interferes with landmark detection and blending, leading to makeup being incorrectly applied to the hair.
- Computational Latency: Many high-quality deep learning models, such as CycleGANs, are too computationally expensive for real-time use on consumer mobile devices.
Proposed Methodology
Our pipeline avoids the heavy overhead of deep learning by utilizing a sophisticated image processing framework consisting of five primary stages:
1. Pre-processing
The source image undergoes whitening and smoothing using OpenCV’s Color Balance and Bilateral Filtering. This creates a "clean canvas" by reducing skin irregularities before the makeup is applied.
2. Facial Alignment
We utilize a modified Active Shape Model (ASM) to identify 90 facial landmark points. An affine transformation then warps the reference image to match the geometric structure of the source face.
3. Layer Decomposition and Transfer
Both images are decomposed into three layers within the CIELAB color space:
- Facial Color Layer: Captures the $a$ and $b$ color channels, which are blended using an alpha-blending approach ($\alpha = 0.95$).
- Facial Detail Layer: Captures fine skin textures and is transferred directly to maintain realism.
- Facial Structure Layer: This layer captures lighting and shading. Our core innovation is a specific Illumination Transfer formula that adjusts the source's structure based on the reference's lighting. This prevents dark makeup from losing depth and becoming gray.
4. Handling Occlusions (Air-Bangs)
To handle hair occlusions, we employ a Fully Convolutional Network (FCN) for facial parsing. This generates a probability-based "soft mask" that distinguishes between skin and hair, ensuring makeup is only applied to skin regions even when partially obscured by hair.
Experimental Results
The robustness of our method was validated using two high-resolution datasets: a "before-makeup" set of 526 images and a "with-makeup" set of 878 images.
Performance and Speed
Our method demonstrates significant efficiency gains over deep learning frameworks. While models like Liu et al. (2016) require approximately 6 seconds on a high-end TITAN X GPU, our algorithm processes a $224 \times 224$ image in under 2 seconds on an iPhone 6.
Qualitative and User Evaluation
By treating illumination as a component of the makeup style, our method successfully transfers dark eyeshadows and foundations that previous state-of-the-art methods (like Guo et al.) failed to render correctly. In a user study conducted via Amazon Mechanical Turk, 87.3% of participants preferred our results over the PairedCycleGAN approach.
Conclusion
This paper introduces a fast, robust, and realistic solution for facial makeup transfer. By combining traditional layer decomposition with a novel illumination transfer technique, we provide a method that handles dark makeup and hair occlusions more effectively than previous approaches. While the system is dependent on accurate landmark detection, its speed and quality make it a superior choice for real-time virtual beauty applications.
Summary updated