Fairness Without Labels: Pseudo-Balancing for Bias Mitigation in Face Gender Classification

Haohua Dong, Ana Manzano Rodríguez, Camille Guinaudeau, Shin'ichi Satoh

Abstract

Automated gender classification systems often exhibit significant demographic bias, particularly against non-Western populations. This paper introduces "pseudo-balancing," a semi-supervised learning (SSL) strategy that leverages unlabeled, diverse datasets to mitigate these biases. By enforcing demographic parity during the pseudo-label selection process, the method significantly reduces performance gaps—most notably narrowing the gender accuracy gap in East Asian populations from over 49% to just 5.01%—without requiring expensive manual annotations.

Introduction

Automated gender classification systems frequently suffer from "demographic blindness." While they may achieve high accuracy on Western populations, performance often plummets when encountering Black or East Asian faces. This disparity typically stems from biased training datasets and algorithms that inadvertently amplify these biases during learning.

While curating perfectly balanced datasets is a common solution, it is prohibitively expensive. Semi-supervised learning (SSL) offers a cost-effective alternative by learning from a small amount of labeled data and a large pool of unlabeled data. However, standard SSL techniques like FixMatch often suffer from the "Matthew effect": the model generates pseudo-labels based on its own biased internal logic, effectively reinforcing its initial prejudices.

The Pseudo-Balancing Method

The authors introduce pseudo-balancing, a lightweight modification to iterative self-training pipelines (specifically FixMatch and FlexMatch). In a standard SSL pipeline, a model predicts labels for unlabeled images; if confident, those images are treated as ground truth for the next round of training. Pseudo-balancing adds a crucial constraint to break the bias feedback loop.

![The pseudo-balancing pipeline:

  1. Pseudo-labeling unlabeled data, 2. Subsampling to balance gender groups, 3. Iterative fine-tuning.](/figures/art_61f325dd5ff643d099eaeae52da8bedf) Source, p. 4

The process involves three key steps:

  1. Partitioning: High-confidence predictions are divided into gender groups.
  2. Subsampling: These groups are manually subsampled to ensure an equal number of "pseudo-male" and "pseudo-female" samples are used in the next training iteration.
  3. Dynamic Adjustment: This works alongside confidence-based selection to prevent the model from being overwhelmed by a majority class, maintaining class equilibrium without requiring ground-truth labels for the secondary dataset.

Experimental Setup and Datasets

The study utilizes three primary datasets to evaluate bias mitigation:

  • Kaggle Gender Classification Dataset: Used to train the initial biased baseline model.
  • FairFace (FF): A demographically balanced dataset used without labels to test if its diversity alone could debias a model.
  • All-Age-Faces (AAF): The primary evaluation benchmark, consisting predominantly of East Asian faces where Western-trained models typically struggle.

Key Results and Discussion

Pseudo-balancing consistently improves fairness while often increasing overall accuracy. The most striking success was observed in the East Asian subgroup, where the baseline model's massive 49% gender gap was narrowed to just 5.01%.

Effect of pseudo-balancing on accuracy and selection rate, showing the best model achieving 79.8% accuracy and a 93.8% selection rate.

Key performance metrics include:

  • Overall Accuracy: Reached 79.81%, a 6.53% improvement over the baseline.
  • Gender Gap Reduction: The overall gender accuracy gap was reduced by 44.17%.
  • Selection Rate (SR): The method achieved a 93.80% Selection Rate, indicating successful bias mitigation (where >80% is the target threshold).

Conclusion and Future Work

Pseudo-balancing demonstrates that you don't need labels to fix a biased model; you just need a diverse, unlabeled "mirror" to reflect a more balanced world. By forcing the model to "slow down" and wait for confident samples from under-represented groups, the method breaks the positive feedback loop of bias.

However, the method relies on the initial model's ability to generate at least some correct pseudo-labels. Future work is needed to address more complex intersectional biases involving age, facial expression, and cultural context, as well as scenarios with extreme initial data skew.

Summary updated

Abstract

Face gender classification models often reflect and amplify demographic biases present in their training data, leading to uneven performance across gender and racial sub-groups. We introduce pseudo-balancing, a simple and effective strategy for mitigating such biases in semi-supervised learning. Our method enforces demographic balance during pseudo-label selection, using only unlabeled images from a race-balanced dataset without requiring access to ground-truth annotations. We evaluate pseudo-balancing under two conditions: (1) fine-tuning a biased gender classifier using unlabeled images from the FairFace dataset, and (2) stress-testing the method with intentionally imbalanced training data to simulate controlled bias scenarios. In both cases, models are evaluated on the All-Age-Faces (AAF) benchmark, which contains a predominantly East Asian population. Our results show that pseudo-balancing consistently improves fairness while preserving or enhancing accuracy. The method achieves 79.81% overall accuracy—a 6.53% improvement over the baseline—and reduces the gender accuracy gap by 44.17%. In the East Asian subgroup, where baseline disparities exceeded 49%, the gap is narrowed to just 5.01%. These findings suggest that even in the absence of label supervision, access to a demographically balanced or moderately skewed unlabeled dataset can serve as a powerful resource for debiasing existing computer vision models.
BibTeX
@misc{dong2025fairness,
  author = {Haohua Dong and Ana Manzano Rodríguez and Camille Guinaudeau and Shin'ichi Satoh},
  title = {{Fairness Without Labels: Pseudo-Balancing for Bias Mitigation in Face Gender Classification}},
  year = {2025},
  eprint = {2510.10191},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.10191},
}
Figure