TL;DR
This paper introduces a novel "w-score" normalization technique for combining scores from multiple recognition systems. By leveraging Extreme Value Theory (EVT) to model the tail of non-match scores, the w-score provides a statistically grounded and robust method for fusion, outperforming traditional z-score normalization, especially when individual recognition systems fail.
What Problem Does This Paper Address?
In many computer vision tasks, combining the outputs of multiple recognition algorithms (fusion) can significantly improve overall performance. Score-level fusion, where the confidence scores from different systems are combined, is a popular approach. However, a critical step before fusion is score normalization, which aims to bring scores from different systems onto a comparable scale. This is challenging because the score distributions of different algorithms can vary widely, and traditional normalization methods often make assumptions about these distributions that may not hold. Furthermore, these methods can be sensitive to "failure" scenarios where one or more recognition systems produce erroneous or misleading scores (e.g., due to adversarial inputs or genuine system limitations). This paper addresses the need for a score normalization technique that is robust to these distributional variations, parameter estimation errors, and, crucially, algorithm failures.
Methods
The core contribution is the w-score normalization technique, which is based on statistical Extreme Value Theory (EVT). Instead of assuming specific distributions for match and non-match scores, the w-score focuses on the tail of the non-match score distribution. The authors argue that the top scores from a recognition system's non-match data, when considered as extrema, naturally follow an Extreme Value Distribution, specifically the Weibull distribution.
The process involves:
- Identifying the tail: For a given recognition instance, the top n scores are selected.
- Fitting a Weibull distribution: A Weibull distribution is fitted to these top scores, excluding the very highest score (hypothesized to be the match) to avoid bias from potential outliers.
- Calculating the w-score: The Cumulative Distribution Function (CDF) of the fitted Weibull distribution is used to transform each score into a probability score. This w-score represents the probability that a given score is not from the non-match distribution, effectively indicating confidence in a match.
- Fusion: These normalized w-scores from multiple algorithms are then combined (e.g., by summing them) to produce a final fused score.
A key advantage highlighted is that this method is adaptive, requiring only scores from a single recognition instance for normalization, and it is robust to noise and algorithm failure. The paper also introduces a mechanism for error detection within the fusion process, identifying potential issues like inaccurate Weibull fitting or data misalignment.
Datasets
The authors evaluated their w-score method on two main types of datasets:
- Biometric Recognition:
- NIST BSSR1 dataset: This dataset includes scores from face recognition (C & G) and fingerprint recognition (LI & RI) algorithms. They used both the multibiometric subset and created a "Chimera" dataset with consistent labeling across algorithms to address limitations of the original set.
- Content-Based Image Retrieval (CBIR):
- Corel "Relevants" set: Contains 50 unique classes.
- INRIA "Holidays" set: Contains 500 unique classes.
They used various CBIR descriptors on these datasets.
Key Insights and Ideas
- EVT for Score Tails: The central insight is that the tail of non-match scores in recognition systems can be effectively modeled by Extreme Value Theory, specifically the Weibull distribution. This provides a strong statistical foundation for normalization, moving beyond ad-hoc methods.
- Robustness to Failure: The w-score's reliance on the tail of non-match scores makes it inherently more robust to individual algorithm failures or impostor scores compared to methods that rely on the entire score distribution. If an algorithm fails, its scores are less likely to influence the Weibull fit of the other algorithms' non-match tails.
- Adaptive Normalization: The w-score is an adaptive technique, meaning it normalizes scores based on the specific instance of recognition, rather than relying on pre-trained models that might not generalize well.
- Built-in Error Detection: The w-score fusion process naturally allows for the detection of potential errors, such as inaccurate model fitting or data misalignment, a property not found in traditional fusion techniques.
- Statistical Basis for Fusion: The w-score transforms raw scores into probabilities of not being a non-match, providing a more interpretable and statistically sound basis for fusion.
Key Results
The paper demonstrates that the w-score normalization consistently outperforms the popular z-score normalization across various biometric and CBIR experiments.
- Biometric Recognition: w-scores achieved significant error reduction compared to z-scores, with improvements ranging from 22.2% to 71.8% in regular fusion scenarios. In failure scenarios (where one algorithm had 0% rank-1 recognition), the advantage was even more pronounced, with error reductions reaching up to 71.8%.
- Content-Based Image Retrieval: Similar improvements were observed, with error reductions ranging from 6.1% to 27.5% in regular fusion and up to 40.3% in failure scenarios.
- Tail Size Sensitivity: The authors note that even small tail sizes (e.g., 3 or 5) for Weibull fitting yielded good normalization, indicating robustness to this parameter.
Limitations and Future Work
- Tail Size Parameter: While robust, the w-score still requires selecting a tail size (n) for fitting the Weibull distribution. The paper suggests that even small values work well, but this remains a parameter to be tuned.
- Assumptions for EVT: The application of EVT relies on assumptions such as the i.i.d. nature of portfolio samples and a sufficient number of classes for the approximation to hold.
- Future Work: The authors suggest further exploration of the error detection capabilities of the w-score and its application to other domains. They also mention that while they focused on Weibull, other EVT distributions (Gumbel, Frechet) could be considered if score distributions are unbounded.
Summary updated