A Deep Insight Into Measuring Face Image Utility With General and Face-Specific Image Quality Metrics

Biying Fu, Cong Chen, Olaf Henniger, Naser DamerWACV 2022

TL;DR

This paper investigates how general image quality assessment (IQA) and face-specific image quality assessment (FIQA) metrics correlate with the utility of face images for facial recognition (FR) systems. The authors found that while deep learning-based FIQA methods generally outperform traditional IQA and handcrafted features, IQA methods still show a significant correlation with FR utility, suggesting potential for hybrid approaches.

What Problem Does This Paper Address?

Facial recognition (FR) systems are increasingly used for authentication, but their performance is heavily dependent on the quality of the input face images. Face Image Quality Assessment (FIQA) aims to predict this "utility" for FR, while general Image Quality Assessment (IQA) focuses on human perception of image quality. A key challenge is understanding the relationship between these two streams of quality assessment and how well general IQA methods can predict face image utility for FR, especially since FIQA methods are often trained with specific FR systems in mind. This paper aims to bridge this gap by systematically evaluating a wide range of IQA and FIQA metrics on their ability to predict face image utility across different FR systems.

Methods

The authors evaluated 25 different quality metrics, categorized into four groups:

  1. General Image Quality Measures (IQA): These include model-based (BRISQUE, NIQE, PIQE), CNN-based (CNNIQA, DeepIQA), multi-task learning-based (MEON, DBCNN), and rank-based (dipIQ, rankIQA, UNIQUE) approaches. These are not specifically designed for face recognition utility.
  2. Handcrafted Image Quality Measures: These are general image quality metrics based on statistical properties of natural images.
  3. Face-Related Handcrafted Measures: These include features derived from spatial domain properties (blur, contrast, luminance, sharpness, etc.) and specific facial features like inter-eye distance, as defined by ISO/IEC TR 29794-5.
  4. Learned Face Utility Measures (DL-based FIQA): These are deep learning models trained to directly assess face image utility. The paper includes six such methods: RankIQ, FaceQnet, SER-FIQ, PFE, MagFace, and SDD-FIQA.

The evaluation involved using these metrics to predict face image utility by observing their impact on the False Non-Match Rate (FNMR) at a fixed False Match Rate (FMR) of 0.1% (FMR1000) across different "reject ratios" (i.e., the percentage of low-quality images discarded).

Datasets

The study utilized three face image databases:

  • BioSecure: A high-quality, controlled database representing passport-like scenarios.
  • Labeled Faces in the Wild (LFW): A widely used benchmark for face verification, containing images from uncontrolled scenarios.
  • VGGFace2: A challenging database with diverse acquisition conditions and quality distributions. A representative subset was used for experiments.

The MTCNN framework was used to detect, align, and crop face images.

Key Insights and Ideas

  • DL-FIQA Dominance: Deep learning-based FIQA methods generally outperform both general IQA and handcrafted features in predicting face image utility, often leading to the lowest FNMR across various reject ratios.
  • IQA's Surprising Utility: Despite not being designed for FR, general IQA methods, particularly model-based and multi-task learning approaches, show a significant and consistent correlation with face image utility. This suggests they can effectively filter out low-quality images that would degrade FR performance.
  • Handcrafted Features Lack Robustness: Individual handcrafted features, like inter-eye distance, showed inconsistent performance across different databases and FR systems. While they can be useful in controlled environments, they are less reliable in diverse, real-world scenarios.
  • Focus Matters: Visualization (using Score-CAM) revealed that DL-based FIQA methods tend to focus their attention on the central facial regions, whereas IQA methods often paid attention to the background as well. This difference in focus might explain the superior performance of FIQA methods.
  • Potential for Hybrid Approaches: The strong correlation of IQA with face utility, combined with the FR model-independent nature of many IQA methods, suggests that integrating IQA with DL-based FIQA could lead to more generalized and robust face utility estimation systems.

Key Results

  • Error Reduction: The study demonstrated that by rejecting a portion of low-quality images based on quality scores, the FNMR could be significantly reduced. For example, on VGGFace2 with ArcFace, CNNIQA reduced FNMR by about 36.7% when rejecting 20-40% of images, while FaceQnet achieved a 19% reduction.
  • Top Performers: For DL-based FIQA, SDD-FIQA and MagFace consistently ranked high. Among IQA methods, CNNIQA and DeepIQA showed strong performance. For handcrafted features, inter-eye distance performed well on VGGFace2 but was less consistent elsewhere.
  • Overlap Analysis: Figure 4 shows that DL-based FIQA methods have high intra-category correlation, and there's a notable inter-category correlation between IQA and DL-based FIQA methods, unlike handcrafted features.

Limitations and Future Work

  • Visualization Limitations: The visualization technique (Score-CAM) was not applicable to all methods, particularly those with complex architectures or patch-based approaches, limiting the scope of visual analysis.
  • FR System Dependency: While DL-based FIQA methods are generally robust, some are trained with specific FR systems, which could influence their performance. The paper highlights the advantage of FR model-independent IQA methods.
  • Future Directions: The authors suggest that combining IQA metrics with DL-based FIQA methods could lead to more generalized measures applicable across different FR systems and application scenarios. Further research into hybrid approaches is warranted.

Summary updated

BibTeX
@inproceedings{fu2022deep,
  author = {Biying Fu and Cong Chen and Olaf Henniger and Naser Damer},
  title = {{A Deep Insight Into Measuring Face Image Utility With General and Face-Specific Image Quality Metrics}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}

Loading related content...

Figure