FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation

Kimmo Kärkkäinen, Jungseock JooWACV 2021 · OpenReview

Abstract

Existing public face image datasets are strongly biased toward Caucasian faces, and other races (e.g., Latino) are significantly underrepresented. The models trained from such datasets suffer from inconsistent classification accuracy, which limits the applicability of face analytic systems to non-White race groups. To mitigate the race bias problem in these datasets, we constructed a novel face image dataset containing 108,501 images which is balanced on race. We define 7 race groups: White, Black, Indian, East Asian, Southeast Asian, Middle Eastern, and Latino. Images were collected from the YFCC-100M Flickr dataset and labeled with race, gender, and age groups. Evaluations were performed on existing face attribute datasets as well as novel image datasets to measure the generalization performance. We find that the model trained from our dataset is substantially more accurate on novel datasets and the accuracy is consistent across race and gender groups.

Introduction: The Challenge of Dataset Bias

Most public face datasets are heavily skewed toward Caucasian individuals. Popular benchmarks like CelebA or LFW are often dominated by White faces, sometimes exceeding 80% representation. When machine learning models are trained on such lopsided data, they develop "asymmetric accuracy"—performing exceptionally well on light-skinned males but failing significantly more often on women and people of color. This bias has real-world ethical consequences, potentially leading to unfair treatment in security, recruitment, and social services.

Methodology

The researchers took a data-centric approach to mitigation, focusing on building a balanced foundation rather than a new algorithm.

Visual samples illustrating the diversity and 'in-the-wild' nature of the images collected compared to traditional biased datasets.
  1. Taxonomy: Seven race groups were defined: White, Black, Indian, East Asian, Southeast Asian, Middle Eastern, and Latino. This taxonomy specifically differentiates between East and Southeast Asian groups and includes Middle Eastern and Latino categories, which are frequently omitted or merged in other datasets.
  2. Collection: Images were primarily sourced from the YFCC-100M Flickr dataset. To avoid "celebrity bias," an adaptive sampling method ensured a more global representation.
  3. Annotation: Amazon Mechanical Turk workers labeled images for race, gender, and age. A "model-in-the-loop" approach was used to re-verify images where human labels and preliminary model predictions disagreed.

Dataset Comparison

The FairFace dataset consists of 108,501 images. Unlike its predecessors, it is specifically designed to be balanced across the seven defined race groups.

Comparison of racial distribution across popular face datasets. FairFace shows a significantly more balanced ratio compared to datasets like CelebA or LFWA+.

As shown in the figure above, traditional datasets like LFWA+ and CelebA are overwhelmingly White. Even newer datasets like DiF (Diversity in Faces) or UTKFace maintain a strong White majority. FairFace is the first large-scale dataset to achieve near-parity across seven racial categories.

Key Results and Generalization

The FairFace model achieved a gender classification accuracy discrepancy of less than 1% between White and non-White groups. In contrast, models trained on other datasets showed gaps as high as 32%. An audit of major commercial APIs (Microsoft, Amazon, IBM, Face++) revealed that while improvements have been made, significant gaps remain, particularly for dark-skinned females, further emphasizing the need for balanced training data.

A key finding is that a model trained on a diverse dataset doesn't just help "minority" groups; it makes the model more robust overall. The FairFace model outperformed others even on novel, "unseen" datasets (Geo-tagged Tweets, Media Photographs, and a Protest Dataset), suggesting that exposure to diverse features helps the model learn more universal facial characteristics.

Conclusion

While race and gender are social constructs and human annotation is inherently subjective, FairFace represents a major step toward reducing algorithmic bias. By providing a balanced, "in-the-wild" dataset, the authors demonstrate that data-centric interventions can significantly improve the fairness and generalization of facial analysis systems.

Summary updated

Abstract

Existing public face image datasets are strongly biased toward Caucasian faces, and other races (e.g., Latino) are significantly underrepresented. The models trained from such datasets suffer from inconsistent classification accuracy, which limits the applicability of face analytic systems to non-White race groups. To mitigate the race bias problem in these datasets, we constructed a novel face image dataset containing 108,501 images which is balanced on race. We define 7 race groups: White, Black, Indian, East Asian, Southeast Asian, Middle Eastern, and Latino. Images were collected from the YFCC-100M Flickr dataset and labeled with race, gender, and age groups. Evaluations were performed on existing face attribute datasets as well as novel image datasets to measure the generalization performance. We find that the model trained from our dataset is substantially more accurate on novel datasets and the accuracy is consistent across race and gender groups. We also compare several commercial computer vision APIs and report their balanced accuracy across gender, race, and age groups. Our code, data, and models are available at https://github.com/joojs/fairface.
BibTeX
@inproceedings{karkkainen2021fairface,
  author = {Kimmo Kärkkäinen and Jungseock Joo},
  title = {{FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=DJvPn1o0TP},
}
Figure