Face parsing and blending-based analysis model for makeup style transfer

XiaoFeng Wang, Yichen Zhang, T. T. Han, Wei Liu, Songling Liu, Yongqin Zhang, Yonghuai LiuResearch Square 2024 · DOI

Introduction

Makeup is a cornerstone of traditional opera culture, carrying deep artistic and aesthetic significance. While contemporary makeup transfer technology has made strides in reproducing subtle styles, it often fails when applied to the intricate, heavy patterns characteristic of traditional Chinese opera (e.g., the Jing or Chou roles). Standard Generative Adversarial Network (GAN) approaches frequently suffer from geometric distortions when facial poses differ and often inadvertently transfer non-makeup elements like headgear or hair accessories.

To address these challenges, this research introduces Opera-MT, a novel makeup transfer model designed to handle complex patterns and textures with high fidelity.

The Opera-MT Framework

The core innovation of Opera-MT lies in its two-stage pipeline that decouples makeup pattern transfer from color style transfer. This separation allows the model to treat pattern migration as a geometric problem and color application as a style-transfer problem.

Architecture of the Opera-MT model, highlighting the decoupling of pattern and color transfer.

1. Makeup Pattern Transfer

This stage ensures that intricate designs—such as a butterfly or specific operatic symbols—are accurately mapped onto the target face without warping.

  • UV Space Transformation: By mapping both the source and reference faces into a 3D UV space, the model establishes a pixel-accurate correspondence. This "pose-invariant" canvas ensures that a pattern on the reference face's cheek is placed exactly on the source face's cheek, regardless of head tilt or rotation.
  • Facial Parsing: Using a pre-trained DML-CSR parser, the model generates masks for specific facial regions. This acts as a filter, ensuring only facial makeup is captured while ignoring distracting elements like hats or background clutter.
  • Poisson Blending: Rather than simple pixel replacement, the model employs Poisson image fusion. This technique blends the gradients of the reference pattern into the source face, preserving fine texture details and creating seamless transitions.

2. Makeup Color Transfer

After the patterns are aligned, an attention-based GAN applies the overall color style (e.g., skin tone and lipstick). By incorporating a cross-attention mechanism and skip connections, the network can focus on specific color features of the reference image while preserving the underlying identity of the source face.

Experimental Evaluation

The effectiveness of Opera-MT was validated using three distinct datasets:

  1. PM Dataset: A new, self-constructed dataset containing 4,400 images, including 1,300 specialized images of the four main Chinese opera roles (Sheng, Dan, Jing, Chou).
  2. MT Dataset: A standard public benchmark for makeup transfer.
  3. Makeup Dataset: A publicly available dataset used in LADN research.

Key Results and Insights

Opera-MT significantly outperformed state-of-the-art baselines like LADN, CPL, and SSAT in both quantitative and qualitative assessments.

  • Quantitative Performance: The model achieved a Multi-Scale Structural Similarity (MS-SSIM) score of 0.929 on the MT dataset, indicating superior structural preservation. It also maintained low Fréchet Inception Distance (FID) scores, demonstrating high-quality image generation and identity preservation.
  • Qualitative Superiority: Visually, Opera-MT is uniquely capable of transferring the sharp, high-contrast lines of "Painted Face" roles without the blurring or distortion common in other models.

Conclusion and Future Directions

By leveraging UV space mapping and facial parsing, Opera-MT overcomes the primary hurdles of heavy makeup transfer. While the model currently faces challenges with extreme facial profiles where occlusion occurs, it represents a significant step forward in digital cultural preservation. Future work aims to integrate this technology with audio and speaker synthesis, potentially allowing for the creation of fully realized digital opera performances.

Summary updated

Abstract

<title>Abstract</title> Makeup is a fundamental element of traditional opera culture, embodying profound artistic connotations and aesthetic values. Incorporating patterns and decorative elements into makeup has also become an essential aspect of contemporary society. Makeup transfer technology enables the reproduction of existing makeup designs onto specified facial images. However, current methodologies face several challenges, including incomplete pattern replication and suboptimal results, particularly in transferring patterned makeup. Consequently, this paper proposes a novel makeup transfer model, termed Opera-MT, which integrates facial parsing with image-fusion techniques. The Opera-MT model first maps facial images onto UV space to establish a pixel-accurate correspondence between the reference face and the source face, effectively addressing issues related to pattern distortion. Subsequently, it integrates facial parsing techniques with Poisson image fusion methods to preserve the texture details of the pattern, thereby overcoming the challenge of incomplete pattern migration in the reference makeup. Finally, the model utilizes a color transfer network to apply the color style of the reference makeup to the source face, resulting in the final output. The effectiveness of Opera-MT has been validated on three datasets, including our collected PM dataset and two publicly available datasets, namely the MT and makeup datasets. The results demonstrate that Opera-MT successfully captures makeup color and pattern texture, producing high-quality, realistic, and natural results. Our code will be available at https://github.com/xiaozhang0927/Opera-MT.
BibTeX
@inproceedings{wang2024face,
  author = {XiaoFeng Wang and Yichen Zhang and T. T. Han and Wei Liu and Songling Liu and Yongqin Zhang and Yonghuai Liu},
  title = {{Face parsing and blending-based analysis model for makeup style transfer}},
  booktitle = {Research Square},
  year = {2024},
  doi = {10.21203/rs.3.rs-5642552/v1},
}
Figure