Exploring Racial Bias within Face Recognition via per-subject Adversarially-Enabled Data Augmentation

Seyma Yucer, Samet Akçay, Noura Al Moubayed, Toby P. Breckon

Abstract

Whilst face recognition applications are becoming increasingly prevalent within our daily lives, leading approaches in the field still suffer from performance bias to the detriment of some racial profiles within society. In this study, we propose a novel adversarial derived data augmentation methodology that aims to enable dataset balance at a per-subject level via the use of image-to-image transformation for the transfer of sensitive racial characteristic facial features. Our aim is to automatically construct a synthesised dataset by transforming facial images across varying racial domains, while still preserving identity-related features, such that racially dependant features subsequently become irrelevant within the determination of subject identity. We construct our experiments on three significant face recognition variants: Softmax, CosFace and ArcFace loss over a common convolutional neural network backbone. In a side-by-side comparison, we show the positive impact our proposed technique can have on the recognition performance for (racial) minority groups within an originally imbalanced training dataset by reducing the pre-race variance in performance.

1. Introduction

Face recognition (FR) systems are increasingly deployed in high-stakes areas such as surveillance, recruitment, and healthcare. However, these systems frequently exhibit significant performance bias, typically performing better on Caucasian faces than on African, Asian, or Indian demographics. This bias is largely rooted in data imbalance; popular training datasets are dominated by Caucasian subjects.

Previous attempts to mitigate this bias often focused on "de-biasing"—attempting to strip racial information from face embeddings. However, racial and identity features are deeply intertwined, and removing one often degrades the other. We propose an alternative: making racial features irrelevant to the model by providing a balanced, synthetic view of every individual across multiple racial domains. By ensuring every subject in a training set is represented across different races, the model is forced to ignore racially dependent features in favor of race-invariant structural identity markers.

2. Methodology

We introduce a per-subject, adversarially-enabled data augmentation pipeline centered on CycleGAN, a framework for unpaired image-to-image translation.

2.1 Race Transfer via CycleGAN

We utilize CycleGAN to learn mappings between four racial domains: African (A), Asian (E), Caucasian (C), and Indian (I). For any given image of a subject, we generate three synthetic versions representing that same individual as if they belonged to the other three racial groups.

CycleGAN-based race transfer and reconstruction process.

2.2 Identity Preservation

A critical component of this pipeline is the cycle-consistency loss inherent in CycleGAN. As shown in the figure above, this loss ensures that while skin tone and certain racial characteristics are transformed (e.g., $x^A \rightarrow x^E$), the underlying geometry and identity-related features remain intact, allowing for a faithful reconstruction ($\hat{x}^A$) of the original subject. This ensures that the augmentation adds racial variety without losing the subject's unique identity.

3. Experimental Setup

Our models utilize a ResNet-101 backbone trained with three state-of-the-art loss functions: Softmax, CosFace, and ArcFace.

  • Training Data: We used BUPT-Transferface to train the CycleGAN models for the 12 possible race-to-race transformations. The FR models were trained on VGGFace2, including a balanced subset of 1,200 subjects (300 per race).
  • Evaluation: Verification accuracy was primarily benchmarked using Racial Faces in-the-Wild (RFW) and Labeled Faces in the Wild (LFW).

4. Results and Discussion

The proposed augmentation method yielded significant improvements in both equity and overall performance:

  • Reduced Performance Variance: The standard deviation (STD) of performance across different racial groups dropped markedly. For instance, using ArcFace on the balanced dataset, the per-race variance decreased from 2.64 to 2.06.
  • Accuracy Gains: Verification performance across all loss functions improved by 0.38% to 1.51%.
  • Minority Group Performance: On the full, imbalanced VGGFace2 dataset, augmenting non-Caucasian subjects specifically narrowed the performance gap between the dominant Caucasian group and minority groups (African and Asian).

These results suggest that bias is not just a matter of subject counts, but of the distribution of features within those groups. Our augmentation method effectively smooths this distribution.

5. Conclusion and Future Work

This study demonstrates that per-subject race transfer is a viable path toward more equitable face recognition. By presenting the model with the same identity across multiple racial "styles," we successfully neutralize sensitive attributes without sacrificing the quality of identity embeddings.

Future work will address limitations such as GAN-induced artifacts in images with poor illumination or extreme poses. We also intend to explore more advanced generative architectures, such as StyleGAN, to improve the realism and robustness of the synthetic transformations.

Summary updated

Abstract

Whilst face recognition applications are becoming increasingly prevalent within our daily lives, leading approaches in the field still suffer from performance bias to the detriment of some racial profiles within society. In this study, we propose a novel adversarial derived data augmentation methodology that aims to enable dataset balance at a per-subject level via the use of image-to-image transformation for the transfer of sensitive racial characteristic facial features. Our aim is to automatically construct a synthesised dataset by transforming facial images across varying racial domains, while still preserving identity-related features, such that racially dependant features subsequently become irrelevant within the determination of subject identity. We construct our experiments on three significant face recognition variants: Softmax, CosFace and ArcFace loss over a common convolutional neural network backbone. In a side-by-side comparison, we show the positive impact our proposed technique can have on the recognition performance for (racial) minority groups within an originally imbalanced training dataset by reducing the pre-race variance in performance.
BibTeX
@inproceedings{yucer2020exploring,
  author = {Seyma Yucer and Samet Akçay and Noura Al Moubayed and Toby P. Breckon},
  title = {{Exploring Racial Bias within Face Recognition via per-subject Adversarially-Enabled Data Augmentation}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  url = {https://openreview.net/forum?id=KhwIKCxeQaL},
}
Figure