A multi-modal dataset, protocol and tools for adaptive biometric systems: a benchmarking study

Ajita Rattani, Gian Luca Marcialis, Fabio RoliInt. J. Biom 2013 · DOI

TL;DR

This paper introduces a novel multi-modal biometric dataset (DIEE) and a standardized protocol to address the lack of comparability in adaptive biometric systems research. By providing a common ground for evaluation, this work aims to enable fair comparisons and independent replication of results, ultimately advancing the field.

What Problem Does This Paper Address?

Adaptive biometric systems, which can update their templates using operational data, promise to overcome the limitations of traditional systems that suffer from performance degradation due to variations in biometric traits over time. However, the field is currently hampered by a lack of standardized evaluation methods. Different research groups use diverse datasets, protocols, and tools, making it impossible to compare their findings. This paper tackles this issue by proposing a comprehensive benchmarking framework—a multi-modal dataset, a unified protocol, and evaluation tools—to facilitate fair and reproducible research in adaptive biometrics.

Methods

The core of this paper lies in its proposed benchmarking framework. The authors introduce:

  • DIEE Multi-modal Dataset: This dataset comprises face and fingerprint modalities collected over 1.5 years from 49 subjects across 10 sessions. Crucially, it maintains the chronological order of samples within sessions and across sessions, capturing both temporary (e.g., pose, illumination) and temporal (e.g., aging) variations. This longitudinal aspect is vital for studying adaptive systems.
  • Unified Protocol: This protocol standardizes key experimental attributes that have previously led to incomparable results. It defines:
  • Training: Using a fixed, small number of initial enrolled templates (two per user from the first session) to simulate realistic limited enrollment scenarios.
  • Updating Threshold: Setting this based on estimated score distributions from the initial templates to balance adaptation and avoid impostor intrusion.
  • Adaptation: Incorporating a realistic scenario with the introduction of random impostor samples into each batch of operational data to test robustness.
  • Performance Assessment: Employing a "joint test and adapt" strategy, where data is used for both testing and adaptation in sequence, and evaluating performance over multiple update cycles.
  • Tools for Performance Evaluation: The paper advocates for specific metrics and analyses, including:
  • Performance Gain Metric: Quantifying the relative improvement of adaptive systems over baseline systems.
  • Analysis of Updated Template Sets: Reporting statistics on captured samples, impostor intrusion, and updated subjects to understand adaptation efficacy.
  • Performance Visualization: Suggesting plots like EER vs. update cycles and EER vs. number of samples to illustrate performance trends over time.

Datasets

The primary dataset introduced and utilized in this paper is the DIEE multi-biometric database.

  • DIEE Multi-biometric Database:
    • Modalities: Face and fingerprint (index and thumb).
  • Subjects: 49.
  • Samples per subject: 100.
  • Time Span: 1.5 years.
  • Sessions: 10.
  • Key Feature: Maintains chronological ordering of samples within and across sessions, capturing both temporary and temporal variations.

The paper also references existing datasets like BANCA, XM2VTS, UMIST, and FVC-DB, but argues that they are not suitable for studying adaptive biometrics due to limitations in sample size, temporal variation capture, or chronological ordering.

Key Insights and Ideas

  • The Criticality of Temporal Variations: The paper strongly emphasizes that to truly evaluate adaptive biometric systems, datasets must capture temporal variations (changes over time, like aging) in addition to temporary variations (changes due to immediate environmental factors). This requires longitudinal data collection over extended periods.
  • Standardization is Key to Progress: The core argument is that without standardized protocols and datasets, progress in adaptive biometrics is hindered by incomparable results. The proposed framework aims to rectify this by providing a common benchmark.
  • Impostor Intrusion is a Realistic Threat: The protocol's inclusion of random impostor samples during adaptation is crucial. It highlights that adaptive systems must be robust to misclassification errors, which can lead to security vulnerabilities.
  • Co-updating Offers Advantages: While not the primary focus of the experimental results, the paper discusses co-updating (where two biometric systems mutually update each other) as a promising approach that can potentially capture more drastic changes than self-updating.
  • Baseline Performance Matters: A high baseline error rate (EER) is presented as an ideal condition for studying adaptation, as it indicates that initial enrolled templates are not representative of operational data, thus necessitating adaptation.

Key Results

The paper presents experimental results using the DIEE dataset and the proposed protocol to evaluate self-updating algorithms for index and thumbprint biometrics.

  • Self-Updating Performance: The experiments show that while self-updating can lead to performance gains (reduction in EER), the gain is not substantial. However, the system tends to stabilize over update cycles.
  • Impostor Intrusion Statistics: For self-updating, the analysis revealed that a small percentage of users were affected by impostor intrusion (around 0.2% for face, 4.08% for fingerprints) at a stringent updating threshold (0.001% FAR). A notable percentage of users also remained un-updated (9.2% to 14.48%).
  • Co-updating vs. Self-updating: Figure 6 illustrates that co-updating (especially when fused with another modality) generally achieves lower Equal Error Rates (EER) compared to self-updating and the baseline, demonstrating enhanced and stable performance over time. This suggests co-updating is more effective in handling variations.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the proposed protocol is not entirely new but rather a refinement and standardization of existing practices.
  • The experimental results presented focus primarily on self-updating, with co-updating shown for comparison but not as extensively analyzed.
  • The study focuses on static image-based biometrics and does not cover video-based adaptation.
  • Future Work:
    • The authors aim for their proposed benchmarking study to facilitate fair comparison and independent replication of results from different research groups.
  • The paper implicitly suggests further exploration of co-updating and other advanced adaptation techniques using the provided dataset and protocol.
  • The development and application of more sophisticated tools for analyzing updated template sets and visualizing performance gains are also implied.

Summary updated

BibTeX
@article{rattani2013multi,
  author = {Ajita Rattani and Gian Luca Marcialis and Fabio Roli},
  title = {{A multi-modal dataset, protocol and tools for adaptive biometric systems: a benchmarking study}},
  journal = {Int. J. Biom},
  year = {2013},
  doi = {10.1504/ijbm.2013.055961},
}
Figure