Exploring Racial Bias within Face Recognition via per-subject Adversarially-Enabled Data Augmentation

Seyma Yucer, Samet Akçay, Noura Al Moubayed, Toby P. Breckon

Abstract

Whilst face recognition applications are becoming increasingly prevalent within our daily lives, leading approaches in the field still suffer from performance bias to the detriment of some racial profiles within society. In this study, we propose a novel adversarial derived data augmentation methodology that aims to enable dataset balance at a per-subject level via the use of image-to-image transformation for the transfer of sensitive racial characteristic facial features. Our aim is to automatically construct a synthesised dataset by transforming facial images across varying racial domains, while still preserving identity-related features, such that racially dependant features subsequently become irrelevant within the determination of subject identity. We construct our experiments on three significant face recognition variants: Softmax, CosFace and ArcFace loss over a common convolutional neural network backbone. In a side-by-side comparison, we show the positive impact our proposed technique can have on the recognition performance for (racial) minority groups within an originally unbalanced training dataset by reducing the per-race variance in performance.

Introduction

Face recognition (FR) systems are increasingly integrated into critical infrastructure, yet they continue to exhibit significant performance disparities across different demographic groups. Specifically, racial minorities often experience higher error rates, a phenomenon largely attributed to the lack of diversity in large-scale training datasets like VGGFace2, which are predominantly Caucasian.

Existing mitigation strategies typically involve adversarial training to suppress sensitive attributes or re-sampling to achieve demographic parity. However, these methods often encounter a "fairness-accuracy trade-off," where the removal of racial information inadvertently degrades identity-specific features. This paper proposes an alternative: rather than removing sensitive information, we augment it at a per-subject level to teach the model that racial characteristics are irrelevant to identity.

Methodology: Per-Subject Adversarial Augmentation

The core of our approach is a data augmentation strategy that generates versions of the same individual across multiple racial domains (African, Asian, Caucasian, and Indian).

Race Transfer via CycleGAN

We utilize CycleGAN to perform unpaired image-to-image translation between four racial groups. By training six CycleGAN models, we facilitate 12 directional mappings. CycleGAN is particularly suited for this task as it does not require paired training data (i.e., photos of the same person in different races) to train the generator.

Figure 1: The race transfer process. The top row illustrates an original African subject ($x^A$) being transformed into an Asian domain ($x^E$) and then reconstructed ($\hat{x}^A$). The bottom row shows an Asian subject ($y^E$) transformed to African ($y^A$) and reconstructed ($\hat{y}^E$).

Identity Preservation

To ensure that the synthetic images retain the subject's unique identity, we leverage the cycle-consistency loss inherent in the CycleGAN architecture. This ensures that while the "style" (racial phenotype) is altered, the "content" (facial geometry and identity markers) remains stable. By providing the Deep Convolutional Neural Network (DCNN) with the same identity across four different racial "packages," the model is forced to ignore skin tone and race-specific features, focusing instead on robust identity markers.

Experimental Setup

We evaluated our approach using a ResNet101 backbone across three state-of-the-art loss functions: Softmax, CosFace, and ArcFace. Training was conducted on several variations of the VGGFace2 dataset, including a baseline balanced subset of 1,200 subjects and an augmented version of the full 8,631-subject dataset. Evaluation was primarily performed on the Racial Faces in-the-Wild (RFW) dataset to measure verification accuracy across demographics.

Results and Discussion

Our experiments yielded several key insights into the nature of bias in FR systems:

  1. Balance is Insufficient: Even with a perfectly balanced dataset, a performance gap of approximately 8% persisted between African and Caucasian groups in baseline tests. This suggests that standard DCNNs are inherently biased toward certain phenotypes regardless of sample size.
  2. Accuracy and Fairness Gains: On the RFW dataset, our augmentation improved verification performance across all loss functions by 0.38% to 1.51%. More importantly, the standard deviation (STD) of performance across races dropped significantly (e.g., from 2.64 to 2.06 for ArcFace), indicating a fairer system.
  3. Recovering Imbalanced Data: Applying this technique to the original imbalanced VGGFace2 dataset improved African group accuracy from 89.45% to 90.10% and reduced overall performance variance from 2.91 to 2.45.

Conclusion

This study demonstrates that per-subject adversarial data augmentation is an effective tool for reducing racial bias in face recognition. By decoupling identity from racial characteristics, we can train more equitable models without sacrificing overall accuracy. Future work will focus on improving the realism of GAN-generated images, particularly under challenging conditions like extreme poses or poor lighting, where current CycleGAN models may introduce artifacts.

Summary updated

Abstract

Whilst face recognition applications are becoming increasingly prevalent within our daily lives, leading approaches in the field still suffer from performance bias to the detriment of some racial profiles within society. In this study, we propose a novel adversarial derived data augmentation methodology that aims to enable dataset balance at a per-subject level via the use of image-to-image transformation for the transfer of sensitive racial characteristic facial features. Our aim is to automatically construct a synthesised dataset by transforming facial images across varying racial domains, while still preserving identity-related features, such that racially dependant features subsequently become irrelevant within the determination of subject identity. We construct our experiments on three significant face recognition variants: Softmax, CosFace and ArcFace loss over a common convolutional neural network backbone. In a side-by-side comparison, we show the positive impact our proposed technique can have on the recognition performance for (racial) minority groups within an originally unbalanced training dataset by reducing the per-race variance in performance.
BibTeX
@inproceedings{yucer2020exploring,
  author = {Seyma Yucer and Samet Akçay and Noura Al Moubayed and Toby P. Breckon},
  title = {{Exploring Racial Bias within Face Recognition via per-subject Adversarially-Enabled Data Augmentation}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  url = {https://openreview.net/forum?id=KhwIKCxeQaL},
}
Figure