Improving Fairness in Facial Albedo Estimation via Visual-Textual Cues

Xingyu Ren, Jiankang Deng, Chao Ma, Yichao Yan, Xiaokang Yang

Abstract

Estimating high-fidelity facial albedo from single "in-the-wild" images is a challenging task due to the inherent ambiguity between skin reflectance and illumination. Traditional methods often suffer from "light-skin bias" and lack fine-grained details. We present ID2Albedo, a framework that leverages the semantic power of the CLIP model to provide visual-textual constraints, effectively mitigating bias and producing high-fidelity 3D face reconstructions across diverse skin tones and ages.

Introduction

In 3D face reconstruction, separating a face's intrinsic color (albedo) from the effects of light and shadow (shading) is a classic ill-posed problem. Historically, models have relied on statistical priors like the Basel Face Model (BFM), which are heavily biased toward light-skinned European phenotypes. Consequently, most existing algorithms struggle to reconstruct people of color accurately, often "whitewashing" them or producing flat, unrealistic textures. Furthermore, traditional PCA-based models lack the capacity to represent fine-grained details like wrinkles and pores.

Methodology

The ID2Albedo framework addresses these challenges through a multi-stage pipeline designed for high fidelity and fairness.

The ID2Albedo pipeline architecture, illustrating the flow from input image to high-fidelity albedo using ArcFace identity features and CLIP-based semantic supervision.

High-Resolution Albedo Generator

Instead of a linear PCA model, we employ a GAN-based generator trained on 142 high-quality 3D facial scans. To overcome dataset size limitations, a "subspace-based" discriminator leverages pre-trained ImageNet features, allowing the model to learn high-frequency details such as pores and wrinkles without overfitting.

Identity-to-Albedo Mapping

The system extracts identity features from an input image using a pre-trained ArcFace network. These features are robust to lighting and pose. A "MappingNet" then translates these identity features into the latent space of the albedo generator.

Visual-Textual Cues and Semantic Loss

To ensure fairness, we use CLIP (a vision-language model) as a semantic supervisor. We pre-define text labels for race, skin tone, age, and wrinkles. CLIP assigns "pseudo-labels" to the input image based on text-image similarity. During training, a differentiable renderer combines the estimated albedo with estimated lighting. We enforce a "Semantic Loss," ensuring that the CLIP-perceived attributes of the reconstructed face match the pseudo-labels of the original input.

Experimental Setup

We evaluated ID2Albedo using several datasets:

  • 3D Scan Store: 142 high-quality albedo maps for training the base generator.
  • SFHQ (Synthetic Faces High Quality): For self-supervised training of the pipeline.
  • FAIR Benchmark: 206 high-quality 3D head scans used for quantitative evaluation of fairness across six skin types (Individual Typology Angle, ITA).
  • FFHQ & CFP: For qualitative "in-the-wild" testing.

Results

ID2Albedo demonstrates significant improvements in both accuracy and fairness compared to state-of-the-art methods.

Qualitative comparison of ID2Albedo against existing methods (TRUST, DECA, Deep3D) on diverse "in-the-wild" subjects, highlighting superior albedo estimation and detail preservation.

Superior Fairness and Accuracy

On the FAIR benchmark, ID2Albedo achieved the best average ITA error (12.07), outperforming TRUST (13.87) and DECA (28.74). Notably, the model showed dramatic improvement in reconstructing very dark skin (Type VI), achieving an ITA score of 10.59.

High Fidelity and Robustness

Qualitatively, ID2Albedo captures subtle textures and "baked-in" features like wrinkles that define a person's actual appearance, avoiding the smooth, plastic look of older methods. The model remains robust under extreme lighting conditions and grayscale inputs, confirming its reliance on identity features rather than pixel-level copying.

Conclusion

ID2Albedo represents a major step forward in unbiased facial reconstruction. By integrating visual-textual cues from CLIP, the framework breaks the ambiguity between skin color and illumination, providing a more equitable and realistic solution for 3D face modeling. Future work will focus on expanding training distributions and handling complex occlusions like glasses or hair.

Summary updated

Abstract

Recent 3D face reconstruction methods have made significant advances in geometry prediction, yet further cosmetic improvements are limited by lagged albedo because inferring albedo from appearance is an ill-posed problem. Although some existing methods consider prior knowledge from illumination to improve albedo estimation, they still produce a light-skin bias due to racially biased albedo models and limited light constraints. In this paper, we reconsider the relationship between albedo and face attributes and propose a ID2Albedo to directly estimate albedo without constraining illumination. Our key insight is that intrinsic semantic attributes such as race, skin color, and age can be used to constrain the albedo map. We first introduce visual-textual cues and design a semantic loss to supervise facial albedo estimation. Specifically, we pre-define text labels such as race, skin color, age, and wrinkles. Then, we employ the text-image model (CLIP) to compute the similarity between the text and the input image, and assign a pseudo-label to each facial image. We constrain generated albedos in the training phase to have the same attributes as the inputs. In addition, we train a high-quality, unbiased facial albedo generator and utilize the semantic loss to learn the mapping from illumination-robust identity features to the albedo latent codes. Finally, our ID2Albedo is trained in a self-supervised way and outperforms state-of-the-art albedo estimation methods in terms of accuracy and fidelity. It is worth mentioning that our approach has excellent generalizability and fairness, especially on in-the- wild data.
BibTeX
@inproceedings{ren2023improving,
  author = {Xingyu Ren and Jiankang Deng and Chao Ma and Yichao Yan and Xiaokang Yang},
  title = {{Improving Fairness in Facial Albedo Estimation via Visual-Textual Cues}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00438},
}

Loading related content...

Figure