SignEEG v1.0: Multimodal Dataset with Electroencephalography and Hand-written Signature for Biometric Systems

Ashish Ranjan Mishra, Rakesh Kumar, Vibha Gupta, Sameer M. Prabhu, Richa Upadhyay, Prakash Chandra Chhipa, Sumit Rakesh, Hamam Mokayed, Debashis Das Chakladar, Kanjar De, Marcus Liwicki, Foteini Liwicki, Rajkumar Saini

TL;DR

This paper introduces SignEEG v1.0, a novel multimodal dataset combining electroencephalography (EEG) and handwritten signatures for enhanced biometric systems. By integrating a forgery-resistant physiological modality (EEG) with a behavioral one (signatures), the dataset aims to significantly improve the robustness and reliability of user identification and verification against sophisticated forgeries.

What Problem Does This Paper Address?

Handwritten signatures are a widely adopted biometric modality due to their convenience and legal recognition. However, they are vulnerable to sophisticated forgery attempts, posing a significant security challenge. To address this, researchers are exploring multimodal approaches that combine signatures with other biometric traits. Electroencephalography (EEG), which captures unique brain activity patterns, is a promising candidate due to its inherent resistance to replication. The challenge lies in effectively integrating these modalities and having accessible, comprehensive datasets to develop and evaluate such systems. Existing research in this area is scarce, often limited to single paradigms, and lacks publicly available data, hindering further investigation.

Methods

The study introduces the SignEEG v1.0 dataset, collected from 70 participants. The data acquisition involved three distinct paradigms:

  1. Mental Imagery (Signature Image): Participants closed their eyes and visualized their signature.
  2. Mental Imagery (Signature Trajectory): Participants closed their eyes and mentally simulated the motor actions of drawing their signature.
  3. Physical Execution: Participants opened their eyes and physically drew their signature on a Wacom tablet.

EEG signals were captured using an Emotiv Insight headset, while signature data was recorded using a Wacom One display pen tablet. The data underwent pre-processing using EEGLAB, including epoching, bandpass filtering, baseline removal, and Independent Component Analysis (ICA) for artifact removal. For evaluation, various machine learning models (Gaussian NB, KNN, Linear SVM, Random Forest, Gradient Boost, XGBoost) were employed, utilizing statistical and wavelet features from EEG, as well as statistical and dynamic features from Wacom signatures.

Datasets

The primary contribution of this paper is the introduction of the SignEEG v1.0 dataset. This dataset comprises:

  • EEG signals and corresponding handwritten signatures.
  • Data from 70 healthy participants.
  • Collected over several months.
  • Includes three distinct experimental paradigms (mental imagery of signature image, mental imagery of signature trajectory, and physical signature execution).
  • Data is available in EDF (for EEG) and CSV (for signatures) formats, along with pre-processed MATLAB structures.
  • The dataset is publicly available on Zenodo under a Creative Commons Attribution 4.0 International License.

Key Insights and Ideas

  • Multimodality as a Forgery Countermeasure: The core insight is that combining a behavioral modality (signatures) with a physiological, forgery-resistant modality (EEG) significantly enhances biometric system robustness against sophisticated forgeries.
  • EEG's Resistance to Replication: EEG signals, representing unique brain activity, are inherently difficult to replicate, making them a strong complement to signatures.
  • Paradigm Diversity is Crucial: The dataset's inclusion of three distinct paradigms (mental imagery and physical execution) allows for a more comprehensive exploration of brain-behavioral correlations in signature-based biometrics.
  • Baseline Performance: The study establishes a baseline performance for various machine learning models, demonstrating the effectiveness of multimodal approaches even with limited sample sizes.

Key Results

The study reports promising results for multimodal biometric systems. For identification, models combining EEG and Wacom signature features achieved high accuracy, with XGBoost reaching 98.94% when using statistical and wavelet EEG features combined with Wacom features. For verification, Random Forest and Gradient Boost achieved the highest accuracy at 94.89% and 93.40% respectively, using a combination of statistical, wavelet, and Wacom features. The results consistently show that multimodal approaches significantly outperform unimodal ones, particularly for signature-based biometrics.

Limitations and Future Work

The authors acknowledge that the longitudinal temporal dynamics of EEG and handwritten signatures are yet to be explored. Future work will focus on intensively studying these temporal dynamics through prolonged, longitudinal data collection from the same subjects. This approach aims to reveal how these biometric modalities evolve and interact over time, potentially leading to further advancements in multimodal biometric system accuracy and reliability. Additionally, the paper notes that data from 5 subjects were set aside for anticipated future research, suggesting potential for further validation or exploration.

Summary updated

Abstract

Handwritten signatures in biometric authentication leverage unique individual characteristics for identification, offering high specificity through dynamic and static properties. However, this modality faces significant challenges from sophisticated forgery attempts, underscoring the need for enhanced security measures in common applications. To address forgery in signature-based biometric systems, integrating a forgery-resistant modality, namely, noninvasive electroencephalography (EEG), which captures unique brain activity patterns, can significantly enhance system robustness by leveraging multimodality's strengths. By combining EEG, a physiological modality, with handwritten signatures, a behavioral modality, our approach capitalizes on the strengths of both, significantly fortifying the robustness of biometric systems through this multimodal integration. In addition, EEG's resistance to replication offers a high-security level, making it a robust addition to user identification and verification. This study presents a new multimodal SignEEG v1.0 dataset based on EEG and hand-drawn signatures from 70 subjects. EEG signals and hand-drawn signatures have been collected with Emotiv Insight and Wacom One sensors, respectively. The multimodal data consists of three paradigms based on mental, & motor imagery, and physical execution: i) thinking of the signature's image, (ii) drawing the signature mentally, and (iii) drawing a signature physically. Extensive experiments have been conducted to establish a baseline with machine learning classifiers. The results demonstrate that multimodality in biometric systems significantly enhances robustness, achieving high reliability even with limited sample sizes. We release the raw, pre-processed data and easy-to-follow implementation details.
BibTeX
@inproceedings{mishra2024signeeg,
  author = {Ashish Ranjan Mishra and Rakesh Kumar and Vibha Gupta and Sameer M. Prabhu and Richa Upadhyay and Prakash Chandra Chhipa and Sumit Rakesh and Hamam Mokayed and Debashis Das Chakladar and Kanjar De and Marcus Liwicki and Foteini Liwicki and Rajkumar Saini},
  title = {{SignEEG v1.0: Multimodal Dataset with Electroencephalography and Hand-written Signature for Biometric Systems}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41597-024-03546-z},
}
Figure