CAST: Conditional Attribute Subsampling Toolkit for Fine-grained Evaluation

Wes Robbins, Steven Zhou, Aman Bhatta, Chad Mello, Vítor Albiero, Kevin W. Bowyer, Terrance E. BoultWACV 2023 · DOI

Introduction

In the field of face recognition (FR), standard benchmarks like LFW (Labeled Faces in the Wild) have long served as the primary yardstick for progress. However, as deep learning models achieve near-perfect accuracy on these datasets, a "ceiling effect" has emerged. High-level accuracy scores often mask significant performance discrepancies across different demographics and image conditions. To address this, we introduce the Conditional Attribute Subsampling Toolkit (CAST), an open-source framework designed for fine-grained, attribute-based evaluation of million-scale datasets.

The CASToolkit Framework

The core of our contribution is a software library that enables researchers to move beyond "single-number" benchmarks. CAST allows for the efficient slicing of massive datasets by over 50 attributes, including race, age, gender, and various image quality metrics.

Overview of the CASToolkit workflow: filtering, subsampling, and evaluation.

The toolkit employs a pre-built metadata index to perform $O(1)$ constant-time filtering. Users can define complex subsets using logical operators—for example, isolating "Female" subjects with "High Quality" images who are "Wearing Glasses." This capability transforms the arduous task of manual data munging into a process that takes seconds. Furthermore, we propose Probabilistic Inter-Category Evaluation, which uses the softmax output of attribute classifiers. This allows images to contribute proportionally to multiple categories, maximizing data utility and providing a more nuanced performance view than traditional disjoint sets.

The C11 Benchmark and the "Easy Pair" Problem

A critical insight from our work is the "Easy Pair" problem. In many standard benchmarks, up to 95% of the computational effort is spent on pairs that are either obviously identical or obviously different. These "easy" pairs inflate accuracy and obscure a model's true discriminative power in challenging scenarios.

To combat this, we introduced C11 (CAST-11), a benchmark composed of 11 sub-benchmarks (including various racial groups, age ranges, and quality levels). Unlike its predecessors, C11 explicitly filters out easy pairs, focusing entirely on the "hard pair" range where cosine similarity falls between 0.15 and 0.30.

Similarity distributions of standard benchmarks vs. the C11 (CC11) benchmark.

As shown in the similarity distributions, standard benchmarks like LFW and CFP-FP exhibit widely separated genuine and imposter distributions. In contrast, the C11 (labeled as CC11 in the figure) benchmark forces the model to operate in the overlap region, where decision thresholds are most critical.

Key Results and Insights

Our evaluation of 12 modern FR architectures revealed several key findings:

  • Benchmark Rigor: While models achieved ~99.8% accuracy on LFW, the average error rate on C11 jumped to 17.93%, highlighting the benchmark's difficulty.
  • The Fairness Gap: Even state-of-the-art models exhibit bias. For instance, the "Southeast Asian Female" cohort consistently yielded the lowest scores, while "Caucasian Male" yielded the highest. This gap expands significantly when evaluated on "hard" pairs.
  • Quality Metrics: We found that general-purpose image quality metrics (e.g., NIMA) are poor predictors of FR performance. However, face-specific metrics like SDD-FIQA are highly effective. Performance tends to saturate quickly; there is a massive accuracy jump from the bottom 10% to the middle 50% of quality, but negligible gains thereafter.

Conclusion and Future Work

CAST provides a powerful tool for identifying hidden biases and failure modes in face recognition systems. While currently focused on FR, the framework is generalizable to other domains like object detection or medical imaging. Future work will focus on using CAST to create balanced training distributions, moving from merely evaluating bias to proactively mitigating it.

The toolkit is publicly available at https://github.com/WesRobbins/CAST.

Summary updated

BibTeX
@inproceedings{robbins2023cast,
  author = {Wes Robbins and Steven Zhou and Aman Bhatta and Chad Mello and Vítor Albiero and Kevin W. Bowyer and Terrance E. Boult},
  title = {{CAST: Conditional Attribute Subsampling Toolkit for Fine-grained Evaluation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00098},
}
Figure