A Quality Aware Sample-to-Sample Comparison for Face Recognition

Mohammad Saeed Ebrahimi Saadabadi, Sahar Rahimi Malakshan, Ali Zafari, Moktari Mostofa, Nasser M. Nasrabadi

TL;DR

This paper introduces QAFace, a novel training method for face recognition that addresses the challenge of imbalanced sample quality in datasets. By adaptively adjusting the learning process to focus on recognizable low-quality samples and ignore unrecognizable ones, QAFace significantly improves face recognition performance, especially on datasets with quality variations.

What Problem Does This Paper Address?

Modern face recognition (FR) systems are trained on massive datasets. However, these datasets often suffer from an imbalance: a large number of high-quality images and a smaller proportion of low-quality images. This imbalance poses a significant problem because standard FR training methods, like the popular Softmax-based classifier, struggle to learn effectively from these underrepresented low-quality samples. This leads to a performance gap between training and testing, particularly when test data exhibits quality variations. Furthermore, current methods often treat all samples equally, inadvertently allowing "unrecognizable" low-quality samples (those even humans struggle to identify) to negatively influence the training of class centers, pulling them away from their optimal positions and hindering overall model generalization.

Methods

The core of QAFace is a "quality-aware injection" mechanism that enhances the standard Softmax-based training. Instead of simply injecting all samples into the class centers, QAFace intelligently selects which samples to use.

Here's how it works:

  • Quality Proxy: The method uses the feature magnitude of a sample as a proxy for its recognizability. Higher feature magnitude generally correlates with higher image quality and recognizability.
  • Adaptive Injection:
    • Recognizable Low-Quality Samples (Hard Samples): These samples, despite being low-quality, are still identifiable. QAFace emphasizes these samples during the injection process, allowing them to contribute more meaningfully to guiding the class centers.
  • Unrecognizable Samples: Samples with very low feature magnitude (below a learned threshold) are deemed unrecognizable and are effectively ignored during the injection phase. This prevents them from distracting the class centers.
  • High-Quality Samples: While important, high-quality samples are given less emphasis during injection because they are already well-represented and their contribution to center updates is less critical compared to hard samples.
  • Normalization: A normalization step is applied to the feature magnitude to make the threshold for ignoring samples consistent across different training stages.
  • Integration: This quality-aware injection is integrated into the existing Softmax framework, complementing the classification signal without requiring significant architectural changes or additional computational overhead for sample selection.

Datasets

Training Data: * Webface4M (4 million samples, ~200,000 identities)

Evaluation Datasets: * CFP-FP (Cross-Pose Frontal-Profile Faces) * LFW (Labeled Faces in the Wild) * CPLFW (Cross-Platform Labeled Faces in the Wild) * CALFW (Cross-Age Labeled Faces in the Wild) * AgeDB * IJB-B (IARPA Janus Benchmark B) * IJB-C (IARPA Janus Benchmark C)

The paper also mentions experiments involving down-sampling of high-quality testing datasets to simulate low-quality scenarios.

Key Insights and Ideas

  • Feature Magnitude as a Quality Indicator: The paper convincingly demonstrates that feature magnitude is a reliable proxy for image quality and recognizability in face images. This is a crucial insight that enables the adaptive sample selection.
  • The Danger of Unrecognizable Samples: The authors highlight how "unrecognizable" low-quality samples can actively harm the training process by misguiding class centers, leading to poorer generalization. This is a key motivation for ignoring them.
  • Focusing on "Hard" but Recognizable Samples: The strategy of emphasizing recognizable low-quality (hard) samples is effective because these samples provide valuable information for improving robustness without the detrimental effects of truly unrecognizable ones.
  • Complementary to Softmax: QAFace doesn't replace Softmax but enhances it. It adds a sample-level quality awareness that Softmax alone lacks, making the overall training more robust.

Key Results

QAFace demonstrates superior performance across multiple benchmarks:

  • Overall Improvement: The proposed method outperforms state-of-the-art algorithms on five out of seven evaluation benchmarks.
  • Specific Dataset Gains: Notably, QAFace shows significant improvements on challenging datasets like IJB-B and IJB-C, which contain diverse image qualities. For instance, on IJB-B, QAFace improves the TAR at FAR=1e-4 compared to VPL.
  • Robustness to Quality Variation: Experiments with down-sampled datasets show that QAFace significantly reduces the performance gap between high-quality and low-quality samples compared to existing methods like ArcFace and VPL.
  • Ablation Studies:
    • The impact of the memory length (At) for storing representations shows optimal performance around 1000 iterations.
  • The threshold (τ) for feature magnitude normalization is found to be critical, with τ=2 yielding the best results on IJB-B and IJB-C.

Limitations and Future Work

Limitations:

  • Saturation on Clean Datasets: On datasets with predominantly high-quality images (like LFW, CFP-FP, etc.), the performance gains are marginal, as these datasets are already close to saturation for current state-of-the-art methods.
  • Dependence on Feature Magnitude: While effective, the reliance on feature magnitude as the sole proxy for quality might be a limitation if other factors (e.g., extreme poses, occlusions) significantly degrade recognizability without affecting feature magnitude in a predictable way.

Future Work:

  • The paper implicitly suggests that further exploration of quality-aware mechanisms could lead to even more robust face recognition systems.
  • Investigating other proxies for image quality or recognizability could be a future direction.
  • Exploring how QAFace might interact with other advanced loss functions or network architectures could yield further improvements.

Summary updated

Abstract

Currently available face datasets mainly consist of a large number of high-quality and a small number of low-quality samples. As a result, a Face Recognition (FR) network fails to learn the distribution of low-quality samples since they are less frequent during training (underrepresented). Moreover, current state-of-the-art FR training paradigms are based on the sample-to-center comparison (i.e., Softmax-based classifier), which results in a lack of uniformity between train and test metrics. This work integrates a quality-aware learning process at the sample level into the classification training paradigm (QAFace). In this regard, Softmax centers are adaptively guided to pay more attention to low-quality samples by using a quality-aware function. Accordingly, QAFace adds a quality-based adjustment to the updating procedure of the Softmax-based classifier to improve the performance on the underrepresented low-quality samples. Our method adaptively finds and assigns more attention to the recognizable low-quality samples in the training datasets. In addition, QAFace ignores the unrecognizable low-quality samples using the feature magnitude as a proxy for quality. As a result, QAFace prevents class centers from getting distracted from the optimal direction. The proposed method is superior to the state-of-the-art algorithms in extensive experimental results on the CFP-FP, LFW, CPLFW, CALFW, AgeDB, IJB-B, and IJB-C datasets.
BibTeX
@inproceedings{saadabadi2023quality,
  author = {Mohammad Saeed Ebrahimi Saadabadi and Sahar Rahimi Malakshan and Ali Zafari and Moktari Mostofa and Nasser M. Nasrabadi},
  title = {{A Quality Aware Sample-to-Sample Comparison for Face Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=GZwK7G7c11},
}
Figure