Abstract
Estimating high-fidelity facial albedo from single "in-the-wild" images is a challenging task due to the inherent ambiguity between skin reflectance and illumination. Traditional methods often suffer from "light-skin bias" and lack fine-grained details. We present ID2Albedo, a framework that leverages the semantic power of the CLIP model to provide visual-textual constraints, effectively mitigating bias and producing high-fidelity 3D face reconstructions across diverse skin tones and ages.
Introduction
In 3D face reconstruction, separating a face's intrinsic color (albedo) from the effects of light and shadow (shading) is a classic ill-posed problem. Historically, models have relied on statistical priors like the Basel Face Model (BFM), which are heavily biased toward light-skinned European phenotypes. Consequently, most existing algorithms struggle to reconstruct people of color accurately, often "whitewashing" them or producing flat, unrealistic textures. Furthermore, traditional PCA-based models lack the capacity to represent fine-grained details like wrinkles and pores.
Methodology
The ID2Albedo framework addresses these challenges through a multi-stage pipeline designed for high fidelity and fairness.
High-Resolution Albedo Generator
Instead of a linear PCA model, we employ a GAN-based generator trained on 142 high-quality 3D facial scans. To overcome dataset size limitations, a "subspace-based" discriminator leverages pre-trained ImageNet features, allowing the model to learn high-frequency details such as pores and wrinkles without overfitting.
Identity-to-Albedo Mapping
The system extracts identity features from an input image using a pre-trained ArcFace network. These features are robust to lighting and pose. A "MappingNet" then translates these identity features into the latent space of the albedo generator.
Visual-Textual Cues and Semantic Loss
To ensure fairness, we use CLIP (a vision-language model) as a semantic supervisor. We pre-define text labels for race, skin tone, age, and wrinkles. CLIP assigns "pseudo-labels" to the input image based on text-image similarity. During training, a differentiable renderer combines the estimated albedo with estimated lighting. We enforce a "Semantic Loss," ensuring that the CLIP-perceived attributes of the reconstructed face match the pseudo-labels of the original input.
Experimental Setup
We evaluated ID2Albedo using several datasets:
- 3D Scan Store: 142 high-quality albedo maps for training the base generator.
- SFHQ (Synthetic Faces High Quality): For self-supervised training of the pipeline.
- FAIR Benchmark: 206 high-quality 3D head scans used for quantitative evaluation of fairness across six skin types (Individual Typology Angle, ITA).
- FFHQ & CFP: For qualitative "in-the-wild" testing.
Results
ID2Albedo demonstrates significant improvements in both accuracy and fairness compared to state-of-the-art methods.
Superior Fairness and Accuracy
On the FAIR benchmark, ID2Albedo achieved the best average ITA error (12.07), outperforming TRUST (13.87) and DECA (28.74). Notably, the model showed dramatic improvement in reconstructing very dark skin (Type VI), achieving an ITA score of 10.59.
High Fidelity and Robustness
Qualitatively, ID2Albedo captures subtle textures and "baked-in" features like wrinkles that define a person's actual appearance, avoiding the smooth, plastic look of older methods. The model remains robust under extreme lighting conditions and grayscale inputs, confirming its reliance on identity features rather than pixel-level copying.
Conclusion
ID2Albedo represents a major step forward in unbiased facial reconstruction. By integrating visual-textual cues from CLIP, the framework breaks the ambiguity between skin color and illumination, providing a more equitable and realistic solution for 3D face modeling. Future work will focus on expanding training distributions and handling complex occlusions like glasses or hair.
Summary updated