Fairness in Face Presentation Attack Detection

Meiling Fang, Wufei Yang, Arjan Kuijper, Vitomir Struc, Naser DamerarXiv 2022 · arXiv

TL;DR

This paper highlights a critical but overlooked gap in biometric security: fairness in Face Presentation Attack Detection (PAD). The authors introduce CAAD-PAD, a multi-attribute annotated dataset, a new metric called Accuracy Balanced Fairness (ABF), and FairSWAP, a data augmentation technique that improves both detection performance and fairness by disrupting identity-related shortcuts in neural networks.

What Problem Does This Paper Address?

While the research community has spent significant effort auditing and mitigating bias in Face Recognition (FR) systems, Face Presentation Attack Detection (PAD)—the "anti-spoofing" layer that prevents attackers from using photos or masks—has been largely ignored.

The core challenge is a lack of data. Most PAD datasets don't include demographic or attribute labels, making it impossible to tell if a system protects all users equally. Preliminary evidence suggests that PAD systems, like FR systems, suffer from "data-induced bias," where models optimize for majority groups (e.g., males without glasses) at the expense of minority groups. This leads to "unfair decisions" where certain demographics are more vulnerable to spoofing attacks than others.

Fig. 1. Aspects effecting fairness in automated decision systems. ∗indicates the addressed aspects in this work, where sampling bias, validation/test data bias, and amplification bias are relevant to the nature of training data, the ODTA on different data groups, and the training and test on group-disjoint data, respectively.

Methods

The authors tackle the fairness problem through three main avenues:

  1. Operational Decision Threshold Assignment (ODTA) Analysis: They investigate how the choice of a decision threshold (the score at which a face is deemed "real") affects different groups. They show that a threshold optimized for a "fused" (mixed) population often results in significantly higher error rates for specific subgroups.
  2. Accuracy Balanced Fairness (ABF) Metric: Existing fairness metrics like the Fairness Discrepancy Rate (FDR) only measure the difference in performance between groups. A model could be perfectly "fair" but have 50% error for everyone. ABF solves this by weighting the fairness discrepancy against the absolute performance of the worst-performing group, ensuring that "fair" doesn't just mean "equally bad."
  3. FairSWAP Augmentation: To mitigate bias, the authors propose a plug-and-play augmentation method. FairSWAP takes attribute-heavy regions (like bangs, beards, or glasses) from one subject and overlays them onto another during training. This forces the model to ignore these "identity shortcuts" and focus instead on the actual artifacts of a presentation attack (e.g., screen glare, print texture).
Visualizing the seven human-annotated attributes in the CAAD-PAD dataset.

Datasets

The authors introduce the Combined Attribute Annotated PAD Dataset (CAAD-PAD). Since no single dataset was large or diverse enough, they combined six existing ones:

  • CASIA-FASD, Replay-Attack, MSU-MFSD, OULU-NPU: Standard datasets for print and replay attacks.
  • HKBU-MARs & SWFFD: Specialized datasets for 3D masks and wax figure attacks.

They manually annotated these with seven binary attributes: Gender, Beard, Eyeglasses, Bangs, Makeup, Hair Length, and Hair Type. This resulted in a benchmark containing 8,190 videos and 4,600 images across 947 subjects.

Key Insights and Ideas

  • The "Identity Shortcut" Hypothesis: The authors suggest that PAD models often fail because they learn to associate specific semantic features (like gender or bangs) with being "bona fide" (real) or "attack." By disrupting these features via FairSWAP, the model is forced to find more robust, low-level attack clues.
  • Thresholding is a Fairness Trigger: Even if a model has similar ROC curves for two groups, the actual deployed fairness depends on the threshold. If the threshold is set based on a male-dominated validation set, female users may experience significantly higher false-alarm or spoof-acceptance rates.
  • Occlusion Vulnerability: Faces with occlusions (beards, glasses, bangs) are consistently less protected across almost all PAD solutions. This suggests that these features interfere with the model's ability to "see" the spoofing artifacts.

Key Results

  • FairSWAP Effectiveness: The FairSWAP method improved both performance (lower Equal Error Rate) and fairness (higher ABF) in 10 out of 12 investigated scenarios.
  • Gender Bias: Baseline models trained on fused data consistently showed lower error rates for males than females. For example, using ResNet50, the EER for males was 2.54% vs. 3.00% for females. FairSWAP reduced these to 1.86% and 2.34%, respectively.
  • Deep Learning vs. Hand-crafted: Deep learning models (ResNet50, DeepPixBis, LMFD-PAD) were found to be generally fairer and more robust than traditional texture-based methods like LBP-MLP, likely because they learn more abstract, hierarchical features.
FairSWAP in action: swapping attribute patches to disorder identity and appearance features.

Limitations and Future Work

  • Attribute Correlation: Some attributes are naturally entangled (e.g., beards and gender). The authors acknowledge that it is difficult to isolate the impact of a single attribute when they are highly correlated in the training data.
  • LBP-MLP Failure: The FairSWAP method did not benefit the LBP-MLP model. The authors hypothesize that the patch-swapping creates illumination discontinuities that degrade the performance of color-channel-based LBP features.
  • Future Directions: The authors suggest exploring more sophisticated bias mitigation strategies beyond data augmentation, such as adversarial training or fairness-aware loss functions, and expanding the dataset to include more diverse racial groups.

Summary updated

Abstract

Face recognition (FR) algorithms have been proven to exhibit discriminatory behaviors against certain demographic and non-demographic groups, raising ethical and legal concerns regarding their deployment in real-world scenarios. Despite the growing number of fairness studies in FR, the fairness of face presentation attack detection (PAD) has been overlooked, mainly due to the lack of appropriately annotated data. To avoid and mitigate the potential negative impact of such behavior, it is essential to assess the fairness in face PAD and develop fair PAD models. To enable fairness analysis in face PAD, we present a Combined Attribute Annotated PAD Dataset (CAAD-PAD), offering seven human-annotated attribute labels. Then, we comprehensively analyze the fairness of PAD and its relation to the nature of the training data and the Operational Decision Threshold Assignment (ODTA) through a set of face PAD solutions. Additionally, we propose a novel metric, the Accuracy Balanced Fairness (ABF), that jointly represents both the PAD fairness and the absolute PAD performance. The experimental results pointed out that female and faces with occluding features (e.g. eyeglasses, beard, etc.) are relatively less protected than male and non-occlusion groups by all PAD solutions. To alleviate this observed unfairness, we propose a plug-and-play data augmentation method, FairSWAP, to disrupt the identity/semantic information and encourage models to mine the attack clues. The extensive experimental results indicate that FairSWAP leads to better-performing and fairer face PADs in 10 out of 12 investigated cases.
BibTeX
@misc{fang2022fairness,
  author = {Meiling Fang and Wufei Yang and Arjan Kuijper and Vitomir Struc and Naser Damer},
  title = {{Fairness in Face Presentation Attack Detection}},
  year = {2022},
  eprint = {2209.09035},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.09035},
}
Figure