Face Verification With Challenging Imposters and Diversified Demographics

Adrian Popescu, Liviu-Daniel Ștefan, Jérôme Deshayes-Chossart, Bogdan IonescuWACV 2022

TL;DR

This paper introduces FaVCI2D, a new face verification dataset designed to address the limitations of existing benchmarks, which often use overly simplistic imposter selection and lack demographic diversity. By creating challenging imposter pairs and ensuring a more balanced demographic representation, FaVCI2D reveals significant performance drops in state-of-the-art models, highlighting that face verification is far from solved and underscoring the need for more realistic evaluation.

What Problem Does This Paper Address?

Face verification (FV) systems, designed to confirm if two images depict the same person, have achieved near-perfect accuracy on many public datasets. However, the authors argue this success is misleading due to two critical flaws in existing datasets:

  1. Oversimplified Imposter Selection: Datasets typically use random selection of identities for imposter pairs. In real-world scenarios, FV systems are more likely to encounter "challenging imposters" – individuals whose faces are visually similar to the target identity. Current datasets don't adequately test this crucial aspect.
  2. Lack of Demographic Diversity: Existing datasets often suffer from imbalanced demographic distributions (gender, age, origin). This can lead to biased performance, where models work well for majority groups but poorly for underrepresented ones, hindering fair and equitable deployment.

These issues not only lead to an inflated sense of performance but also raise ethical concerns and can contribute to the withdrawal of datasets due to public pressure.

Methods

The core contribution of this paper is the creation of the FaVCI2D dataset, built with specific design choices to overcome the aforementioned problems:

  • Challenging Imposter Pairs: Instead of random selection, imposter pairs are generated by using a deep face recognition model to find visually similar identities. These pairs are then manually verified to ensure the identities are indeed different.
  • Challenging Genuine Pairs: To further test model robustness, genuine pairs are created by selecting visually distinct images of the same identity.
  • Demographic Diversification: The dataset aims for a balanced representation across gender, geographic origin (country), and age. This is achieved by carefully filtering and selecting identities from Wikipedia, leveraging metadata and employing classifiers for gender and origin.
  • Data Minimization and Legal Compliance: The dataset adheres to GDPR principles by storing only necessary information (two images per identity) and anonymizing identities. It also prioritizes using resources with suitable licenses to ensure long-term sustainability and avoid legal issues.
  • Visual Prototype and Pair Validation: A deep learning pipeline (MTCNN for detection, ir50 for feature extraction) is used to create visual prototypes for each identity. This aids annotators in the manual verification process for both genuine and imposter pairs.

Datasets

  • Existing Datasets Used for Evaluation: LFW [21], YTF [37], IJB-C [24], MegaFace [19], TrillionPairs [3], DiF [25], FairFace [18], RFW [36].
  • Newly Introduced Benchmark: FaVCI2D (Face Verification with Challenging Imposters and Diversified Demographics). This dataset is constructed from publicly available resources, primarily Wikipedia, and is designed to be more challenging and demographically representative than existing benchmarks.

Key Insights and Ideas

  • "Near 100% Accuracy" is Misleading: The paper strongly argues that current high performance on standard datasets is an artifact of their simplified evaluation settings, particularly the random selection of imposters.
  • Challenging Imposters Reveal True Model Limits: Introducing visually similar imposters leads to a significant performance drop across all tested state-of-the-art models, demonstrating that face verification is still a challenging task.
  • Demographic Bias is Deeply Rooted: The performance differences observed across genders and origins are often linked to the demographic imbalances present in the large-scale datasets used to train the underlying face recognition models.
  • Country as a Proxy for Origin is More Robust: The authors found that using country of origin provides a more objective and less controversial segmentation compared to race or skin color, aligning with ethical considerations.
  • Dataset Sustainability is Crucial: The paper emphasizes the importance of legal and ethical considerations (licensing, data protection) for the long-term viability of datasets, citing past withdrawals as cautionary tales.

Key Results

  • Significant Performance Drop: State-of-the-art models that achieved near 100% accuracy on existing datasets showed a substantial performance decrease on FaVCI2D, especially when using challenging imposter pairs. For instance, the "insightface" model, which performed at 95.75% with challenging imposters and a large pool size, dropped to 82.61% when the imposter similarity was set to 1 (most similar).
  • Gender Bias: Accuracy was generally lower for female face verification, with the gap widening for more challenging imposter settings. This is attributed to gender imbalance in training datasets.
  • Origin Bias: Performance varied by country of origin, with lower accuracy for regions less represented in training data (e.g., Tunisia, Japan, South Korea). Europe and North America generally showed higher performance, reflecting their overrepresentation in training datasets.
  • Age-Related Performance: Models showed slightly lower performance for extreme age ranges (18-25 and 76-100), suggesting potential bias against younger and older individuals.

Limitations and Future Work

  • Africa's Underrepresentation: Despite efforts, achieving perfect demographic balance for Africa was challenging due to a scarcity of reusable data. The authors plan to release revised versions of the dataset to address this.
  • Imposter Pool Size: While FaVCI2D offers a large imposter pool, the authors note that an even larger pool might be beneficial for further evaluating performance, though this would exacerbate demographic imbalance.
  • Future Work: The authors suggest that FaVCI2D can be used to calibrate future face recognition datasets for better demographic coverage. They also highlight the need for further research into gender bias, particularly with models trained on balanced datasets.

Summary updated

BibTeX
@inproceedings{popescu2022face,
  author = {Adrian Popescu and Liviu-Daniel Ștefan and Jérôme Deshayes-Chossart and Bogdan Ionescu},
  title = {{Face Verification With Challenging Imposters and Diversified Demographics}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure