Abstract
Racial bias is an important issue in biometric, but has not been thoroughly studied in deep face recognition. In this paper, we first contribute a dedicated dataset called Racial Faces in-the-Wild (RFW) database, on which we firmly validated the racial bias of four commercial APIs and four state-of-the-art (SOTA) algorithms. Then, we further present the solution using deep unsupervised domain adaptation and propose a deep information maximization adaptation network (IMAN) to alleviate this bias by using Caucasian as source domain and other races as target domains. This unsupervised method simultaneously aligns global distribution to decrease race gap at domain-level, and learns the discriminative target representations at cluster level. A novel mutual information loss is proposed to further enhance the discriminative ability of network output without label information. Extensive experiments on RFW, GBU, and IJB-A databases show that IMAN successfully learns features that generalize well across different races and across different databases.
Introduction
Deep learning has revolutionized face recognition (FR), yet it harbors a significant "dark secret": racial bias. Most state-of-the-art (SOTA) models are trained on datasets where Caucasians constitute 75-85% of the data. Consequently, these models perform significantly worse on non-Caucasian faces—with error rates sometimes twice as high for African faces compared to Caucasian ones.
Addressing this via Unsupervised Domain Adaptation (UDA) presents unique challenges. Unlike standard object recognition where classes are consistent across domains, FR involves entirely different identities in the source (e.g., Caucasian) and target (e.g., African) domains. Furthermore, the model must learn fine-grained features to distinguish individuals within a new race, rather than just aligning the overall "look" of the domains.
The Racial Faces in-the-Wild (RFW) Dataset
To benchmark and mitigate this bias, the authors introduced the Racial Faces in-the-Wild (RFW) database. It contains four subsets—Caucasian, Asian, Indian, and African—with 3,000 individuals and approximately 10,000 images per subset. Crucially, RFW is balanced for pose, age, and gender to ensure that bias measurements specifically target racial differences.
Proposed Method: Information Maximization Adaptation Network (IMAN)
The authors propose IMAN, a three-stage framework designed to transfer knowledge from a labeled source race to an unlabeled target race:
- Global Alignment (MMD): The model uses Maximum Mean Discrepancy (MMD) to minimize the statistical distance between the feature distributions of the source and target races, providing a baseline alignment.
- Pseudo-Labeling via Clustering: Since target identities are unknown, IMAN uses spectral clustering to group similar target faces and assigns "pseudo-labels." The network is then pre-adapted using these labels with a standard Softmax loss.
- Mutual Information (MI) Loss: This core novelty maximizes the information between unlabeled target data and the network's predictions. This encourages "confident" and "diverse" predictions, effectively pushing decision margins apart in an unsupervised manner.
Experimental Results
Experiments on RFW validated that SOTA algorithms (like ArcFace) and commercial APIs consistently perform best on Caucasians and worst on African and Asian faces. IMAN-A (using ArcFace as a base) significantly improved verification accuracy on African faces from 85.13% to 91.42% and on Asian faces from 86.27% to 91.15%.
Key insights include:
- Algorithmic Bias: Performance gaps remain even with race-balanced training data, suggesting that current feature extraction algorithms may inherently struggle with certain skin tones or facial features.
- Regional Developer Bias: Commercial APIs developed in East Asia performed better on Asian faces, while Western-developed APIs performed better on Caucasians, highlighting how regional data bias mirrors developer bias.
- Generalization: IMAN also improved performance on the "Ugly" partition of the GBU dataset (from 75% to 85.38%), proving it can adapt to domain gaps caused by lighting and image quality, not just race.
Conclusion
IMAN successfully narrows the racial performance gap in face recognition through unsupervised adaptation. While it does not eliminate the gap entirely—darker skin tones remain more difficult to preprocess and extract features from due to lighting and contrast issues—it provides a powerful framework for learning discriminative features across different races without target-race labels. Future work may involve specialized network architectures or loss functions targeting the unique geometry and texture features of different races.
Summary updated