Score Normalization for Demographic Fairness in Face Recognition

Yu Linghu, Tiago de Freitas Pereira, Christophe Ecabert, Sébastien Marcel, Manuel Günther

TL;DR

This paper demonstrates that demographic bias in face recognition can be significantly mitigated through score normalization—a post-processing step that requires no retraining of the underlying AI models. By aligning the score distributions of different demographic groups using "cohort" data, the researchers improved fairness across gender and ethnicity without sacrificing overall verification accuracy.

What Problem Does This Paper Address?

Modern face recognition (FR) systems are notoriously biased. Due to imbalances in training data (which often skew toward white male faces), these systems frequently exhibit the "Other Race Effect," resulting in higher False Match Rates (FMR) or False Non-Match Rates (FNMR) for women and people of color.

The standard industry approach to fixing this usually involves "fairness-aware" training or fine-tuning, which is computationally expensive, requires massive labeled datasets, and is impossible if you are using a proprietary "black box" system. The authors tackle a different question: Can we fix fairness at the very end of the pipeline by simply adjusting the similarity scores?

Methods

The authors explore several score normalization techniques that adjust the raw similarity score ($s$) between two faces based on how that score compares to a "cohort" (a reference set of unrelated faces).

  1. Identity-based (Z-norm/T-norm): Standard techniques that normalize scores based on a specific subject’s typical similarity to a general cohort.
  2. Demographics-based (M1.1 - M2.2): The authors extend Z/T-norms by restricting the cohort to the same demographic group as the person being verified.
  3. Pure Cohort-based (M3): Instead of looking at individual subjects, this method uses the mean and standard deviation of all impostor scores within a specific demographic group to shift the distribution.
  4. Genuine and Impostor Cohorts (M4 & M5): Most methods only look at "impostor" scores (how much two different people look alike). Methods M4 (Platt Scaling) and M5 (Bimodal-CDF) incorporate "genuine" scores (how much two images of the same person look alike) to better align both ends of the distribution.

Datasets

The study utilizes two major benchmarks:

  • VGGFace2: Used to evaluate both gender and ethnicity bias.
  • RFW (Racial Faces in the Wild): A standard for testing ethnic bias.
  • Random RFW (New Protocol): The authors identified a selection bias in the original RFW protocol (which uses "difficult" impostors). They introduced a new "random" protocol to provide a more realistic evaluation of how these systems perform in the wild.

Key Insights and Ideas

  • Post-processing is Powerful: You don't always need to retrain a model to make it fairer. Normalizing scores is "model-agnostic," meaning it can be applied to any existing face recognition system.
  • Demographics Matter More than Identity: Normalizing based on demographic groups (e.g., "how does this score compare to other Asian faces?") was more effective at reducing bias than normalizing based on the specific individual's identity.
  • The Sub-sampling Trap: A surprising finding was that "balancing" a dataset through sub-sampling (removing samples from the majority group) actually worsened fairness. This is because it reduces the data available for the system to learn the distribution of the minority group, amplifying existing errors.
  • The FMR/FNMR Trade-off: Fairness isn't just about one number. The authors used the WERM (Worst-case Error Rate) metric to ensure that improvements in False Matches didn't come at the cost of a massive increase in False Non-Matches (people being wrongly locked out of their own accounts).

Key Results

The researchers tested their methods on five state-of-the-art pre-trained networks (including ArcFace and MagFace).

  • Fairness Gains: In almost all cases, demographic-based normalization (M1.1, M1.2, and M3) led to a less biased output. For example, on the VGGFace2 ethnicity task, the WERM (lower is better) dropped from 3.81 to 1.52 for the E1 network.
  • Stability: Unlike some training-based methods, these normalization techniques rarely "downgraded" the system's accuracy. The True Match Rate (TMR) remained stable or even improved slightly in several experiments.
  • M3 is the "Sweet Spot": The "Pure Cohort-based" method (M3) proved to be highly effective and computationally efficient because the statistics can be pre-computed during the enrollment phase.

Limitations and Future Work

The authors acknowledge that their current models assume score distributions follow a Normal (Gaussian) distribution. In reality, the "tails" of these distributions (where the most critical errors happen) might follow different patterns.

Future work will involve exploring other probability distributions to better capture these tail behaviors. Additionally, the authors plan to refine the WERM metric to better account for the unequal contributions of FMR and FNMR in different real-world security scenarios.

Summary updated

Abstract

Fair biometric algorithms have similar verification performance across different demographic groups given a single decision threshold. Unfortunately, for state-of-the-art face recognition networks, score distributions differ between demographics. Contrary to work that tries to align those distributions by extra training or fine-tuning, we solely focus on score post-processing methods. As proved, well-known sample-centered score normalization techniques, Z-norm and T-norm, do not improve fairness for high-security operating points. Thus, we extend the standard Z/T-norm to integrate demographic information in normalization. Additionally, we investigate several possibilities to incorporate cohort similarities for both genuine and impostor pairs per demographic to improve fairness across different operating points. We run experiments on two datasets with different demographics (gender and ethnicity) and show that our techniques generally improve the overall fairness of five state-of-the-art pre-trained face recognition networks, without downgrading verification performance. We also indicate that an equal contribution of False Match Rate (FMR) and False Non-Match Rate (FNMR) in fairness evaluation is required for the highest gains. Code and protocols are available.
BibTeX
@inproceedings{linghu2024score,
  author = {Yu Linghu and Tiago de Freitas Pereira and Christophe Ecabert and Sébastien Marcel and Manuel Günther},
  title = {{Score Normalization for Demographic Fairness in Face Recognition}},
  booktitle = {IJCB},
  year = {2024},
  doi = {10.1109/ijcb62174.2024.10744514},
}

Loading related content...

Figure