Efficient Explainable Face Verification Based on Similarity Score Argument Backpropagation

Marco Huber, Anh Thi Luu, Philipp Terhörst, Naser DamerWACV 2024

TL;DR

This paper introduces xSSAB, an efficient, training-free method for explaining face verification decisions by backpropagating similarity score arguments. They also present Patch-LFW, a novel benchmark and evaluation protocol to quantitatively assess the quality of these explanations, demonstrating xSSAB's superior trade-off between efficiency and performance compared to state-of-the-art methods.

What Problem Does This Paper Address?

As face recognition (FR) systems become increasingly prevalent in security-critical applications, understanding why a system matches or rejects a pair of faces is crucial for building trust, ensuring accountability, and identifying potential biases. Current deep learning-based FR models are often "black boxes" due to their complexity. While existing explainability methods exist, they often require extensive training, are computationally expensive, or lack rigorous quantitative evaluation, making it difficult to compare their effectiveness. This paper aims to bridge this gap by providing an efficient, white-box approach for generating interpretable explanations and a standardized way to evaluate them.

Methods

The core of the proposed method, xSSAB (explainable Similarity Score Argument Backpropagation), works by extending a standard Siamese FR network with a cosine similarity layer. This layer calculates the cosine similarity between face embeddings. The key innovation lies in how explanations are generated:

  • Similarity Score Argumentation: The method defines an "argument threshold" based on the system's decision threshold. Features contributing more strongly than this threshold are considered "positive arguments" (supporting a match), while those below are "negative arguments" (supporting a non-match).
  • Gradient Backpropagation: Gradients are then backpropagated through the network, but crucially, they are selectively calculated based on either the positive or negative arguments. This allows the method to highlight pixels that contribute to either a match or a non-match decision.
  • Map Fusion: Two explanation maps (one for similarity, one for dissimilarity) are generated and then combined into a single, unified map. A Gaussian blur is applied for smoother visualization.
  • Training-Free: A significant advantage is that xSSAB is training-free, meaning it can be applied to any pre-trained FR model without requiring additional training or fine-tuning.

Datasets

  • Training: The paper mentions that the FR models used were trained on the MS1M-V2 dataset.
  • Evaluation:
    • Patch-LFW: A novel benchmark dataset introduced by the authors. It is based on the Labeled Faces in the Wild (LFW) dataset. To artificially increase "false matches" (FM) and "false non-matches" (FNM), patches from different identities or the same identity are added to genuine and imposter pairs, respectively. This manipulation is designed to create scenarios where FR systems are more likely to make errors, thus providing a better testbed for explainability methods.
  • LFW: Used for determining the Equal Error Rate (EER) threshold for the Patch-LFW evaluation protocol.

Key Insights and Ideas

  • Argument-based Explanation: Shifting from purely visual saliency to analyzing the arguments (positive/negative contributions) of features to the similarity score provides a more nuanced understanding of the decision-making process.
  • Efficiency through White-Box Approach: By leveraging backpropagation on an extended FR model, xSSAB achieves high efficiency without the computational burden of black-box methods that require multiple input perturbations.
  • Quantitative Evaluation is Crucial: The introduction of Patch-LFW and the Decision-based Patch Replacement (DPR) curve evaluation protocol addresses a critical gap in the field, enabling objective comparison of explainability methods.
  • Generalizability: The training-free nature of xSSAB allows it to be applied across a variety of SOTA FR models, demonstrating its broad applicability.

Key Results

  • Qualitative: xSSAB generates fine-grained explanation maps that visually highlight similar and dissimilar areas, often more detailed than those from competing methods like xFace and xCos.
  • Quantitative (Patch-LFW DPR):
    • xSSAB outperforms the random baseline significantly on both FMR and FNMR across various FR models.
  • On the ArcFace model, xSSAB is competitive with the trained xCosw approach and superior to the trained xCoss approach.
  • While black-box xFace methods show slightly better performance in reducing errors, they are orders of magnitude slower.
  • Efficiency (Latency): xSSAB is significantly faster than the xFace methods, requiring only one forward and one backward pass per image pair. It is also faster than the trained xCos approach, despite being training-free. For example, xSSAB takes approximately 0.24s per explanation map, compared to ~12s for xFace methods.

Limitations and Future Work

  • Limitations:
    • The Patch-LFW dataset is artificially created, and the ability of explainability methods to identify non-artificial triggers of verification decisions needs further investigation.
  • While xSSAB outperforms xCos on some models, the pre-trained xCos module did not perform competitively on other models, suggesting that training the xCos module is often necessary for good performance.
  • Future Work: The paper implicitly suggests that future work should focus on evaluating explainability methods on real-world scenarios with non-artificial biases and occlusions. The authors also highlight the ongoing importance of explainability, interpretability, and transparency in biometric systems due to the sensitive nature of the data processed.

Summary updated

BibTeX
@inproceedings{huber2024efficient,
  author = {Marco Huber and Anh Thi Luu and Philipp Terhörst and Naser Damer},
  title = {{Efficient Explainable Face Verification Based on Similarity Score Argument Backpropagation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure