Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition

Xingchao Yang, Takafumi Taketomi, Yoshihiro Kanamori

Introduction

In the realm of 3D digital human creation, achieving realistic facial makeup remains a significant hurdle. Traditionally, artists in the visual effects (VFX) and gaming industries have relied on labor-intensive manual texture painting or expensive, specialized "light stage" capture systems to acquire high-fidelity makeup data. While 2D makeup transfer techniques have emerged as a more accessible alternative, they suffer from a fundamental flaw: they "bake" the lighting conditions of the source photograph into the result. This entanglement of lighting and texture prevents the transferred makeup from reacting naturally to new virtual environments.

Presented at Eurographics 2023, this research introduces a novel pipeline designed to "unbake" makeup from a single, unconstrained 2D portrait. By decomposing an image into its constituent components—geometry, skin albedo, lighting, and makeup—the authors enable the extraction of illumination-independent makeup representations in 3D UV space.

The Challenge of Illumination and Pose

Existing 2D methods often fail when faced with complex real-world conditions. If a source image contains harsh shadows or specular highlights, these artifacts are typically transferred along with the makeup, leading to unrealistic results. Furthermore, 2D methods struggle with variations in head pose and occlusions, such as hair or hands covering parts of the face.

To address these issues, this paper proposes a transition from 2D pixel-based manipulation to a 3D UV representation. By mapping the face to a standardized 3D coordinate system, the method ensures that makeup remains spatially consistent regardless of the subject's pose or expression. Crucially, the system explicitly models and removes lighting effects before extraction, resulting in a "flat" makeup texture ready for relighting.

Methodology: A Coarse-to-Fine Decomposition

The researchers utilize a three-stage process to disentangle makeup from the underlying facial attributes:

1. Coarse 3D Reconstruction

The pipeline begins by employing a 3D Morphable Model (3DMM), specifically the FLAME model, to estimate the basic 3D shape and initial textures. A key innovation here is the extension of standard techniques to estimate not only diffuse lighting but also specular shading (the physical "shininess" of the skin). The authors also implement a skin-tone adjustment layer to correct for inherent biases often found in 3DMM texture priors.

2. UV Completion and Refinement

Because a single 2D photo only provides a partial view of the face, the projected UV map contains missing regions. The authors use a specialized Deep Spatial Decomposition GAN (DSD-GAN) to "inpaint" these occluded or invisible areas. Following completion, an optimization loop refines the texture to capture fine-grained details of the skin and lighting that the initial 3DMM might have missed.

3. Neural Makeup Extraction

The core of the method lies in treating makeup as a semi-transparent layer governed by alpha blending logic. A dedicated neural network (Network E) decomposes the refined UV texture into three distinct components:

  • Bare Skin Albedo: The underlying appearance of the skin without makeup.
  • Makeup Albedo: The pure color of the makeup products.
  • Alpha Matte: A transparency map defining the thickness and intensity of the makeup at every pixel.

This decomposition follows the formula: $Image = Alpha \cdot Skin + (1 - Alpha) \cdot Makeup$, allowing for precise control over the final appearance.

Data and the Parametric Makeup Model

The model was trained on a combination of massive facial datasets (FFHQ and CelebA-HQ) for UV completion and specialized makeup datasets (MT and LADN). By processing these through their pipeline, the authors generated a new large-scale dataset of 60,073 high-quality makeup textures in UV space.

Leveraging this data, the researchers constructed the first parametric statistical model for facial makeup using Principal Component Analysis (PCA). Much like how 3DMMs allow users to modify facial structure with a few sliders, this model enables the generation of entirely new, realistic makeup styles by navigating a low-dimensional "makeup space."

Results and Applications

The proposed method demonstrates several advantages over previous state-of-the-art techniques like BeautyGAN or PSGAN:

  • Relightable Transfer: Makeup can be extracted from a shadowed source and applied to a target in a completely different lighting environment without artifacts.
  • Intuitive Editing: By manipulating the alpha matte, users can perform smooth "makeup removal" or "application" animations, or adjust the intensity of specific elements like lipstick or eyeshadow.
  • Quantitative Superiority: Ablation studies confirmed that the inclusion of specular reconstruction significantly improved perceptual similarity (LPIPS) and structural accuracy (SSIM) in the final 3D reconstructions.

Limitations and Future Work

While the method represents a significant leap forward, some limitations remain. The underlying 3DMM priors still contain some "baked-in" lighting from their original capture sessions, which can occasionally influence the refinement step. Additionally, the current model primarily focuses on the diffuse properties of makeup; complex specular effects, such as glitter or high-gloss lip finishes, are not yet fully captured. Future research may also explore how heavy makeup or prosthetics physically alter the 3D geometry of the skin surface, beyond just changing its color and transparency.

Summary updated

BibTeX
@inproceedings{yang2023makeup,
  author = {Xingchao Yang and Takafumi Taketomi and Yoshihiro Kanamori},
  title = {{Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition}},
  booktitle = {Computer Graphics Forum (Proc. of Eurographics )},
  year = {2023},
  url = {https://openreview.net/forum?id=IWq5PCG9hT},
}
Figure