Abstract
Makeup transfer aims to transplant makeup from a reference face to a source face while preserving the source's identity. While existing methods have made progress, they often struggle with color fidelity and spatial transformation. We propose the Facial Attribute Transformer (FAT) and its variant, Spatial FAT, to address these issues. By modeling semantic correspondences and integrating Thin Plate Splines (TPS), our method achieves high-fidelity color transfer and, for the first time, geometric attribute transformation.
Introduction: Beyond Color Overlays
Traditional makeup transfer models often treat the process as a simple 2D color overlay. However, makeup frequently alters the perceived geometry of facial features—for instance, an eyebrow pencil can reshape an arch, or contouring can redefine bone structure. Existing state-of-the-art models often fail to capture these subtle geometric shifts and specific color shades (color fidelity). Furthermore, many attention-based models are computationally inefficient due to sequential processing.
The Facial Attribute Transformer (FAT)
The core of our approach is the Facial Attribute Transformer (FAT). Inspired by Transformer architectures in Natural Language Processing, FAT utilizes a mutual-attention mechanism to establish semantic correspondences between the source and reference faces. Instead of a simple pixel-to-pixel mapping, the model learns the relationship between facial components (e.g., mapping reference "lip" features to the source "lip" area).
To enhance robustness against varying poses and expressions, we introduce Landmark Embedding (LE). Rather than standard positional encodings, we use the distance from each pixel to 68 facial landmarks, providing the model with a precise spatial map of the face.
Spatial FAT and Geometric Transfer
To handle the geometric aspects of makeup, we developed Spatial FAT. This extension integrates Thin Plate Splines (TPS), a flexible warping technique, into the Transformer framework. This allows the model to predict and apply local spatial transformations, effectively warping reference attributes like eyebrow shapes to fit the source face's landmarks.
Training with Pseudo Ground Truth
A significant challenge in makeup transfer is the lack of paired training data (the same person with and without makeup in identical poses). We address this by generating Pseudo Ground Truth (PGT). By warping the reference face onto the source face's geometry, we create high-quality training targets that allow the model to learn precise attribute distillation.
Results and Performance
Extensive experiments on the MT (Makeup Transfer) and M-Wild datasets demonstrate that FAT outperforms previous methods in both realism and accuracy.
Key Advantages:
- Color Fidelity: FAT captures specific shades and textures (e.g., matte vs. gloss) that other models often wash out.
- Efficiency: By processing facial parts in parallel rather than sequentially, FAT is significantly faster than predecessors. On a GPU, FAT achieves an inference speed of ~22ms, roughly 5x to 7x faster than previous state-of-the-art models like PSGAN.
- High Resolution: Through a pyramid reconstruction technique, the framework supports the generation of crisp 1k-resolution images.
Conclusion
Facial Attribute Transformers represent a significant step forward in makeup transfer by treating shape as a transferable attribute alongside color and texture. By combining attention mechanisms with TPS-based warping, FAT provides a robust, precise, and efficient solution for high-fidelity facial attribute manipulation. Future work will explore handling extreme occlusions and extending this framework to other domains like virtual clothing try-on.
Summary updated