Ensemble-Based Bounding Box Regression for Enhanced Knuckle Localization

Ritesh Vyas, Bryan M. Williams, Hossein Rahmani, Ricki Boswell-Challand, Zheheng Jiang, Plamen Angelov, Sue Black

TL;DR

This paper introduces an ensemble approach for automatically locating knuckle regions on human hands, crucial for forensic identification when other biometrics fail. By combining multiple object detection models, they achieve significantly improved accuracy and robustness compared to individual detectors and existing methods, particularly in challenging, unconstrained scenarios.

What Problem Does This Paper Address?

Forensic science heavily relies on biometrics like fingerprints and facial recognition. However, these can be unavailable or of poor quality in crime scenes. Hand-based biometrics, specifically knuckle creases, offer a promising alternative. The challenge lies in accurately and automatically localizing these knuckle regions (MCP, PIP, and DIP joints) from images, which can vary greatly in background, lighting, finger pose, and even accessories like rings. Existing methods often require specific hand poses or are limited in their evaluation, making them less suitable for real-world forensic applications.

Methods

The core of this work is an ensemble-based bounding box regression approach. Instead of relying on a single object detector, the authors combine the strengths of multiple state-of-the-art deep learning models. They utilize both single-stage detectors (YOLO-v2 and YOLO-v3) for speed and two-stage detectors (Faster R-CNN with ResNet50 and ResNet101 backbones) for accuracy. These individual predictions are then fused using Weighted Box Fusion (WBF), a technique that combines bounding boxes from multiple detectors, weighted by their confidence scores, to produce a more robust and accurate final prediction. This ensemble strategy aims to overcome the limitations of individual detectors and improve generalizability.

Datasets

The study utilizes three datasets:

  • HD Dataset: A public dataset with 4650 hand-dorsal images from 501 subjects, featuring open fingers and a consistent pose.
  • 11k Dataset: A larger public dataset with 11,076 hand images (5680 dorsal views) from 190 subjects, exhibiting varying hand poses, finger positions, and orientations.
  • H-Unique (HUQ) Dataset: A novel, unconstrained, multi-ethnic hand dorsal dataset collected by the authors, featuring significant variations in backgrounds, illuminations, and orientations, designed to mimic real-world crime scene conditions.

A subset of 1000 images from each dataset was used to create ground truth annotations for training and evaluation.

Key Insights and Ideas

  • Ensemble Power: Combining multiple object detectors significantly outperforms individual models, especially in challenging, unconstrained environments. This highlights the benefit of leveraging diverse model strengths.
  • WBF Superiority: Weighted Box Fusion proves to be an effective method for merging predictions, leading to more accurate and inclusive bounding boxes compared to traditional non-maximal suppression.
  • Robustness in Unconstrained Scenarios: The proposed method demonstrates remarkable performance even on the H-Unique dataset, which mimics real-world forensic scenarios with significant variations, suggesting its practical applicability.
  • Beyond Localization: The paper introduces novel metrics for evaluating knuckle localization suitability for identification, moving beyond simple overlap measures.

Key Results

The proposed ensemble approach consistently outperformed individual detectors and the state-of-the-art Openpose keypoint detector across various metrics and datasets.

  • Suitability for Identification: The ensemble achieved over 99% suitability for PIP and DIP knuckles on the HD and 11k datasets, and 96.39% and 97.82% on the challenging HUQ dataset, respectively.
  • Mean IoU: The ensemble achieved the highest mean IoU values across all knuckle types and datasets, indicating superior bounding box fitting. For example, on the HD dataset, the mean IoU for major knuckles was 0.7702, significantly higher than other methods.
  • Point-Form Analysis (QBB Accuracy): The ensemble achieved the highest Quarter Bounding Box (QBB) accuracy, the most stringent metric, demonstrating precise centroid localization. For instance, on the HD dataset, the WBF approach achieved 52.85% QBB accuracy, far exceeding Openpose's 0.25%.
  • Matching Performance: The ensemble approach also led to improved biometric matching performance, with mean EER values of 5.26% for HD and 8.07% for 11k datasets, and high mean AUC values (up to 99.04% for HD).

Limitations and Future Work

  • Data Scarcity: The authors acknowledge that data scarcity is a common issue in biometrics, which their ensemble approach helps mitigate.
  • MCP Knuckle Prominence: The prominence of creases in MCP knuckles is less than in PIP and DIP, leading to slightly lower suitability scores for MCPs.
  • Future Work: The authors plan to investigate knuckle localization in the 3D domain and extend their approach to a larger number of unconstrained images with greater variations in illumination, orientation, and background. They also suggest exploring Graph Convolutional Networks (GCNs) and Transformers for future refinements.

Summary updated

BibTeX
@inproceedings{vyas2022ensemble,
  author = {Ritesh Vyas and Bryan M. Williams and Hossein Rahmani and Ricki Boswell-Challand and Zheheng Jiang and Plamen Angelov and Sue Black},
  title = {{Ensemble-Based Bounding Box Regression for Enhanced Knuckle Localization}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=ktfpldN2Y9Q},
}
Figure