Lipstick ain't enough: Beyond Color Matching for In-the-Wild Makeup Transfer

Thao Nguyen, Anh Tran, Minh Hoai

Abstract

Traditional makeup transfer methods often struggle with "extreme" makeup styles, such as stickers, face paintings, and jewels, typically treating the task as simple color manipulation. This paper introduces a holistic framework that leverages 3D UV space for precise facial alignment and a dual-branch architecture to independently process color and complex patterns. By decoupling these elements, the proposed method significantly outperforms state-of-the-art techniques on both light and extreme makeup styles, as demonstrated on the new Color and Pattern Makeup (CPM) dataset.

Introduction

Most existing makeup transfer methods, such as BeautyGAN or PSGAN, are designed for subtle color adjustments like lipstick or eyeshadow. However, they frequently fail when applied to "in-the-wild" scenarios involving intricate designs like festival glitter or face decals. A primary challenge is facial misalignment; variations in pose or expression between source and reference images often lead to distorted or misplaced makeup. The authors argue that a robust transfer system must account for the 3D structure of the face to handle both global color shifts and localized pattern additions accurately.

Example of "extreme" makeup involving complex patterns and stickers.

Methodology

The proposed framework utilizes a UV map representation to solve the alignment problem. By employing a pre-trained 3D face model (PRNet), the system "unfolds" the face into a flattened 2D texture map. This ensures that every pixel corresponds to a fixed semantic point on the face, regardless of the original head pose.

The architecture consists of two parallel branches:

  1. Color Transfer Branch: This branch handles skin foundation, lip color, and eyeshadow using a CycleGAN-like structure. It employs an improved Histogram Matching loss optimized for aligned UV textures.
  2. Pattern Transfer Branch: A novel U-Net-based segmentation network is used to detect and extract discrete makeup patterns, such as stickers or drawings.
The framework distinguishes between localized patterns (e.g., butterflies) and color-based makeup (e.g., lipstick).

The outputs from both branches are blended and re-wrapped onto the source image's 3D face model for the final result.

Datasets and Evaluation

To address the lack of extreme makeup data, the researchers introduced the Color and Pattern Makeup (CPM) datasets:

  • CPM-Real: 3,895 real-world images with diverse styles for testing.
  • CPM-Synt-1 & 2: Synthetic datasets used for training the pattern segmentation module and evaluating transfer performance.

The model was also benchmarked against the standard MT (Makeup Transfer) dataset for traditional styles.

Key Results and Insights

  • Superior Fidelity: Qualitative comparisons show the method accurately transfers complex stickers while maintaining skin texture.
  • Quantitative Performance: On the CPM-Synt-2 dataset, the model achieved an MS-SSIM of 0.977, far exceeding previous methods like LADN (0.656).
  • User Preference: In a survey of 40 participants, the method scored 4.60/5.0 on the CPM-Real dataset, significantly higher than the next best method, PSGAN (2.76).
  • Controllability: The modular design allows for partial transfers, enabling users to mix elements (e.g., lipstick from one reference and patterns from another).

Conclusion

By moving beyond simple color matching and incorporating 3D facial geometry, this framework sets a new standard for "in-the-wild" makeup transfer. While there is room to improve fine-grained pattern segmentation and handle occlusions, the decoupling of color and pattern represents a significant step forward in realistic facial augmentation.

Summary updated

BibTeX
@inproceedings{nguyen2021lipstick,
  author = {Thao Nguyen and Anh Tran and Minh Hoai},
  title = {{Lipstick ain't enough: Beyond Color Matching for In-the-Wild Makeup Transfer}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=wmSbyut2czg},
}
Figure