Face Quality Estimation and Its Correlation to Demographic and Non-Demographic Bias in Face Recognition

Philipp Terhörst, Jan Niklas Kolf, Naser Damer, Florian Kirchbuchner, Arjan KuijperarXiv 2020 · arXiv

TL;DR

This paper reveals that Face Quality Assessment (FQA) algorithms—tools designed to filter out "bad" images to improve recognition—unintentionally inherit and amplify the demographic biases of the face recognition (FR) systems they support. By analyzing state-of-the-art FQA methods, the authors demonstrate that subgroups already marginalized by FR systems (such as specific ethnicities and age groups) are systematically assigned lower quality scores, potentially leading to discriminatory outcomes during enrollment and authentication.

What Problem Does This Paper Address?

Modern face recognition systems are highly accurate but notoriously biased, often performing significantly worse on certain demographic groups (e.g., non-white ethnicities, young children) or under specific conditions like profile poses. To boost performance, researchers use FQA to estimate an image's "utility"—how useful it is for a specific recognition model.

The prevailing wisdom is that FQA should be "model-specific" to be most effective. However, the authors identify a critical gap: if an FQA algorithm is tailored to a biased FR model, does it also become biased? This "bias transfer" could lead to a "double whammy" for marginalized groups: not only is the system less likely to recognize them correctly, but it might also reject their images entirely during enrollment, labeling them as "low quality" simply because the underlying model struggles to process their features.

Methods

The researchers conducted a comprehensive audit of the relationship between FR bias and FQA bias. They evaluated four prominent FQA solutions:

  1. COTS: A commercial off-the-shelf industry product.
  2. Best-Rowden: An SVM-based approach trained on comparison scores.
  3. FaceQnet: A deep learning regressor trained to predict quality labels.
  4. SER-FIQ: A recent unsupervised method that measures quality based on the "robustness" of embeddings under stochastic variations (dropout).

These were tested against two widely used FR models, FaceNet and ArcFace, across two datasets representing controlled and uncontrolled environments. The primary evaluation tool was the Error-versus-Reject Curve (ERC), which measures how much the verification error drops as the lowest-quality images are discarded.

Datasets

  • ColorFERET: A controlled dataset of 14,000 images from 1,200 individuals, featuring labels for pose, ethnicity, and age.
  • Adience: An uncontrolled, "in-the-wild" dataset of 26,500 images from 2,200 individuals, primarily used for age and gender analysis.
  • Training Data: The FQA models were variously trained or evaluated using MORPH, VGGFace2, and MS1M to ensure a broad demographic spread.

Key Insights and Ideas

  • Quality is Utility, Not Just Aesthetics: The paper emphasizes that "quality" in biometrics isn't about how pretty a picture is; it's about how well a specific algorithm can use it. If a model is "blind" to certain features, it perceives images with those features as low utility.
  • The Bias Transfer Effect: The most effective FQA methods (like SER-FIQ) are the ones most tightly coupled to the FR model. Ironically, this tight coupling makes them the most prone to bias transfer.
  • Embeddings Over Training Data: The authors found that the bias in FQA originates more from the underlying FR embeddings than from the specific data used to train the FQA regressor itself.
  • Ethical Implications: Biased FQA creates systemic barriers. For example, in automated border control, a biased FQA might constantly flag a specific demographic for manual inspection, not because their photos are blurry, but because the system is poorly trained on their demographic.

Key Results

The study confirmed significant biases in the underlying FR models. For instance, FaceNet showed a staggering 80.02% False Non-Match Rate (FNMR) for children aged 0-2, compared to much lower rates for adults. Profile poses resulted in error rates 10 times higher than frontal views.

When applying FQA thresholds, the bias transfer became clear:

  • Subgroup Attrition: As the quality threshold increases, the proportion of "biased" subgroups (e.g., Black individuals, young children) in the remaining pool drops sharply. This indicates the FQA is disproportionately labeling these groups as "low quality."
  • Performance vs. Fairness: While SER-FIQ provided the best quality estimation performance (see Figure 1), it also exhibited the clearest separation in quality distributions between biased and unbiased groups.
Figure 1: Face quality assessment performance on the ColorFeret and Adience datasets using two face embeddings, FaceNet and Arcface. The FNMR is reported at a FMR of 0.1%.
Figure 2: Analysis of FaceNet embeddings showing how the proportion of different subgroups (pose, ethnicity, age) changes as low-quality images are rejected. Note the sharp decline in certain groups.

Limitations and Future Work

The authors acknowledge that the current definition of biometric quality—utility for recognition—is fundamentally at odds with demographic fairness if the recognition models themselves are biased.

Future directions include: * Developing FQA solutions that are "demographic-blind" or specifically designed to resist bias transfer. * Prioritizing the development of unbiased FR models, which would naturally lead to more equitable model-specific FQA. * Re-evaluating the ethical frameworks and standards (like ISO/IEC) that define biometric quality to include fairness as a core metric.

Summary updated

Abstract

Face quality assessment aims at estimating the utility of a face image for the purpose of recognition. It is a key factor to achieve high face recognition performances. Currently, the high performance of these face recognition systems come with the cost of a strong bias against demographic and non-demographic sub-groups. Recent work has shown that face quality assessment algorithms should adapt to the deployed face recognition system, in order to achieve highly accurate and robust quality estimations. However, this could lead to a bias transfer towards the face quality assessment leading to discriminatory effects e.g. during enrolment. In this work, we present an in-depth analysis of the correlation between bias in face recognition and face quality assessment. Experiments were conducted on two publicly available datasets captured under controlled and uncontrolled circumstances with two popular face embeddings. We evaluated four state-of-the-art solutions for face quality assessment towards biases to pose, ethnicity, and age. The experiments showed that the face quality assessment solutions assign significantly lower quality values towards subgroups affected by the recognition bias demonstrating that these approaches are biased as well. This raises ethical questions towards fairness and discrimination which future works have to address.
BibTeX
@misc{terhorst2020face,
  author = {Philipp Terhörst and Jan Niklas Kolf and Naser Damer and Florian Kirchbuchner and Arjan Kuijper},
  title = {{Face Quality Estimation and Its Correlation to Demographic and Non-Demographic Bias in Face Recognition}},
  year = {2020},
  eprint = {2004.01019},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.01019},
}
Figure