Facial Attribute Transformers for Precise and Robust Makeup Transfer

Zhaoyi Wan, Haoran Chen, Jie An, Wentao Jiang, Cong Yao, Jiebo Luo

Abstract

Makeup transfer aims to transplant makeup from a reference face to a source face while preserving the source's identity. While existing methods have made progress, they often struggle with color fidelity and spatial transformation. We propose the Facial Attribute Transformer (FAT) and its variant, Spatial FAT, to address these issues. By modeling semantic correspondences and integrating Thin Plate Splines (TPS), our method achieves high-fidelity color transfer and, for the first time, geometric attribute transformation.

Introduction: Beyond Color Overlays

Traditional makeup transfer models often treat the process as a simple 2D color overlay. However, makeup frequently alters the perceived geometry of facial features—for instance, an eyebrow pencil can reshape an arch, or contouring can redefine bone structure. Existing state-of-the-art models often fail to capture these subtle geometric shifts and specific color shades (color fidelity). Furthermore, many attention-based models are computationally inefficient due to sequential processing.

The Facial Attribute Transformer (FAT)

The core of our approach is the Facial Attribute Transformer (FAT). Inspired by Transformer architectures in Natural Language Processing, FAT utilizes a mutual-attention mechanism to establish semantic correspondences between the source and reference faces. Instead of a simple pixel-to-pixel mapping, the model learns the relationship between facial components (e.g., mapping reference "lip" features to the source "lip" area).

To enhance robustness against varying poses and expressions, we introduce Landmark Embedding (LE). Rather than standard positional encodings, we use the distance from each pixel to 68 facial landmarks, providing the model with a precise spatial map of the face.

Spatial FAT and Geometric Transfer

To handle the geometric aspects of makeup, we developed Spatial FAT. This extension integrates Thin Plate Splines (TPS), a flexible warping technique, into the Transformer framework. This allows the model to predict and apply local spatial transformations, effectively warping reference attributes like eyebrow shapes to fit the source face's landmarks.

Spatial FAT enables the precise transfer of geometric attributes, such as the shape and arch of eyebrows, as seen in this detailed eye-area transfer.

Training with Pseudo Ground Truth

A significant challenge in makeup transfer is the lack of paired training data (the same person with and without makeup in identical poses). We address this by generating Pseudo Ground Truth (PGT). By warping the reference face onto the source face's geometry, we create high-quality training targets that allow the model to learn precise attribute distillation.

Results and Performance

Extensive experiments on the MT (Makeup Transfer) and M-Wild datasets demonstrate that FAT outperforms previous methods in both realism and accuracy.

FAT achieves exceptional color fidelity and texture preservation, capturing subtle details like lip gloss sheen and specific lipstick shades.

Key Advantages:

  • Color Fidelity: FAT captures specific shades and textures (e.g., matte vs. gloss) that other models often wash out.
  • Efficiency: By processing facial parts in parallel rather than sequentially, FAT is significantly faster than predecessors. On a GPU, FAT achieves an inference speed of ~22ms, roughly 5x to 7x faster than previous state-of-the-art models like PSGAN.
  • High Resolution: Through a pyramid reconstruction technique, the framework supports the generation of crisp 1k-resolution images.

Conclusion

Facial Attribute Transformers represent a significant step forward in makeup transfer by treating shape as a transferable attribute alongside color and texture. By combining attention mechanisms with TPS-based warping, FAT provides a robust, precise, and efficient solution for high-fidelity facial attribute manipulation. Future work will explore handling extreme occlusions and extending this framework to other domains like virtual clothing try-on.

Summary updated

Abstract

In this paper, we address the problem of makeup transfer, which aims at transplanting the makeup from the reference face to the source face while preserving the identity of the source. Existing makeup transfer methods have made notable progress in generating realistic makeup faces, but do not perform well in terms of color fidelity and spatial transformation. To tackle these issues, we propose a novel Facial Attribute Transformer (FAT) and its variant Spatial FAT for high-quality makeup transfer. Drawing inspirations from the Transformer in NLP, FAT is able to model the semantic correspondences and interactions between the source face and reference face, and then precisely estimate and transfer the facial attributes. To further facilitate shape deformation and transformation of facial parts, we also integrate thin plate splines (TPS) into FAT, thus creating Spatial FAT, which is the first method that can transfer geometric attributes in addition to color and texture. Extensive qualitative and quantitative experiments demonstrate the effectiveness and superiority of our proposed FATs in the following aspects: (1) ensuring high-fidelity color transfer; (2) allowing for geometric transformation of facial parts; (3) handling facial variations (such as poses and shadows) and (4) supporting high-resolution face generation.
BibTeX
@inproceedings{wan2022facial,
  author = {Zhaoyi Wan and Haoran Chen and Jie An and Wentao Jiang and Cong Yao and Jiebo Luo},
  title = {{Facial Attribute Transformers for Precise and Robust Makeup Transfer}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=ta1pOXGA5Py},
}
Figure