Racial Faces in the Wild: Reducing Racial Bias by Information Maximization Adaptation Network

Mei Wang, Weihong Deng, Jiani Hu, Xunqiang Tao, Yaohai Huang

Abstract

Racial bias is an important issue in biometrics but has not been thoroughly studied in deep face recognition. In this paper, we first contribute a dedicated dataset called Racial Faces in-the-Wild (RFW) database, on which we firmly validated the racial bias of four commercial APIs and four state-of-the-art (SOTA) algorithms. Then, we further present the solution using deep unsupervised domain adaptation and propose a deep information maximization adaptation network (IMAN) to alleviate this bias by using Caucasian as source domain and other races as target domains. This unsupervised method simultaneously aligns global distribution to decrease race gap at domain-level, and learns the discriminative target representations at cluster level. A novel mutual information loss is proposed to further enhance the discriminative ability of network output without label information. Extensive experiments on RFW, GBU, and IJB-A databases show that IMAN successfully learns features that generalize well across different races and across different databases.

Introduction

Deep learning has revolutionized face recognition (FR), yet it harbors a significant "dark secret": racial bias. Most popular training datasets are heavily skewed toward Caucasian faces, often exceeding 80% representation. This imbalance leads to models that perform significantly worse on non-Caucasian individuals. For instance, Amazon’s Rekognition tool famously misidentified 28 U.S. congressmen as criminal mugshots, with error rates approaching 40% for non-Caucasian individuals.

Addressing this bias is hindered by two primary challenges. First, there is a lack of diverse benchmarks; existing test sets like LFW or IJB-A do not possess sufficient racial diversity to accurately measure bias. Second, standard Unsupervised Domain Adaptation (UDA) techniques—designed to adapt models from a source domain (e.g., Caucasian faces) to a target domain (e.g., African faces)—typically assume shared identities between domains. In face recognition, identities in the training and test sets are entirely disjoint, causing standard UDA methods to fail.

The Racial Faces in-the-Wild (RFW) Dataset

To address the lack of benchmarks, the authors introduced the Racial Faces in-the-Wild (RFW) database. It consists of four subsets—Caucasian, Asian, Indian, and African—each containing approximately 10,000 images of 3,000 individuals. This balanced dataset allows for a rigorous evaluation of racial bias across different systems.

Examples of faces and mean faces from the four racial subsets in the RFW dataset: Caucasian, Indian, Asian, and African.

Initial experiments on RFW revealed a "double error" reality: for nearly all commercial APIs and state-of-the-art algorithms, the error rate for African faces is roughly double that of Caucasian faces. Interestingly, geographic bias was also evident; APIs developed by East Asian companies performed better on Asian faces than their Western counterparts, confirming that the origin of training data and algorithms influences bias.

Information Maximization Adaptation Network (IMAN)

The proposed solution, IMAN, treats different races as distinct domains and employs unsupervised domain adaptation to bridge the performance gap without requiring new labeled data for target races.

Positive and negative face pairs from the RFW dataset used to evaluate verification performance across different races.

IMAN utilizes three core mechanisms:

  1. Global Alignment: Using Maximum Mean Discrepancy (MMD), the model aligns the high-level feature distributions of the source (Caucasian) and target (other races) domains.
  2. Clustering-based Pseudo-labels: Since target domain labels are unavailable, the model uses spectral clustering to group similar faces into "pseudo-identities," enabling preliminary supervised learning.
  3. Mutual Information (MI) Loss: This novel loss function enhances discriminative ability by minimizing conditional entropy (increasing prediction certainty for individual faces) and maximizing marginal entropy (ensuring a diverse distribution of predictions to avoid model collapse).

Experimental Results

IMAN demonstrates significant improvements in narrowing the racial performance gap. When using ArcFace as a base (IMAN-A), verification accuracy on African faces jumped from 85.13% to 91.42%. IMAN consistently outperformed other domain adaptation methods like DDC and DAN by approximately 3% across all non-Caucasian test sets.

Beyond race, the method proved versatile. On the "Ugly" partition of the GBU dataset—characterized by difficult lighting and poses—IMAN improved performance from 75% to 85.38%, demonstrating its robustness to various types of domain gaps.

Conclusion

While IMAN significantly reduces racial bias, it does not achieve perfect parity. Darker skin tones still present physical challenges for feature extraction in current camera pipelines due to low contrast. Future research will focus on specialized feature extractors for different skin tones and exploring cooperative relationships between races during training to further improve global recognition fairness.

Summary updated

Abstract

Racial bias is an important issue in biometric, but has not been thoroughly studied in deep face recognition. In this paper, we first contribute a dedicated dataset called Racial Faces in-the-Wild (RFW) database, on which we firmly validated the racial bias of four commercial APIs and four state-of-the-art (SOTA) algorithms. Then, we further present the solution using deep unsupervised domain adaptation and propose a deep information maximization adaptation network (IMAN) to alleviate this bias by using Caucasian as source domain and other races as target domains. This unsupervised method simultaneously aligns global distribution to decrease race gap at domain-level, and learns the discriminative target representations at cluster level. A novel mutual information loss is proposed to further enhance the discriminative ability of network output without label information. Extensive experiments on RFW, GBU, and IJB-A databases show that IMAN successfully learns features that generalize well across different races and across different databases.
BibTeX
@inproceedings{wang2019racial,
  author = {Mei Wang and Weihong Deng and Jiani Hu and Xunqiang Tao and Yaohai Huang},
  title = {{Racial Faces in the Wild: Reducing Racial Bias by Information Maximization Adaptation Network}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=B6v-PCoGuRF},
}
Figure