SignEEG v1.0 : Multimodal Dataset with Electroencephalography and Hand-written Signature for Biometric Systems

Ashish Ranjan Mishra, Rakesh Kumar, Vibha Gupta, Sameer M. Prabhu, Richa Upadhyay, Prakash Chandra Chhipa, Sumit Rakesh, Hamam Mokayed, Debashis Das Chakladar, Kanjar De, Marcus Liwicki, Foteini Liwicki, Rajkumar Saini

TL;DR

This paper introduces SignEEG v1.0, a novel multimodal dataset combining electroencephalography (EEG) and handwritten signatures to enhance biometric system robustness against sophisticated forgeries. By integrating physiological (EEG) and behavioral (signature) modalities, the authors demonstrate significantly improved identification and verification accuracy, paving the way for more secure and reliable biometric solutions.

What Problem Does This Paper Address?

Handwritten signatures are a widely used biometric modality due to their convenience and legal recognition. However, they are highly susceptible to forgery attempts, which can be difficult to detect. To combat this, researchers are exploring multimodal approaches that combine signatures with other, more forgery-resistant modalities. Electroencephalography (EEG), which captures unique brain activity patterns, is a promising candidate due to its inherent difficulty to replicate. The challenge lies in effectively integrating these two modalities and creating a comprehensive dataset that supports research in this area. Existing work in this domain is limited by scarce data and a lack of standardized experimental paradigms.

Methods

The study proposes a multimodal biometric framework that combines EEG and handwritten signatures. The core of their contribution is the creation of the SignEEG v1.0 dataset, which includes data from 70 participants. The data collection involved three distinct experimental paradigms:

  1. Mental Imagery (Signature Image): Participants close their eyes and visualize their signature.
  2. Mental Imagery (Signature Trajectory): Participants close their eyes and mentally simulate the motor movements required to draw their signature.
  3. Physical Execution: Participants physically draw their signature on a Wacom tablet with their eyes open.

EEG data was acquired using an Emotiv Insight headset, and signature data was captured using a Wacom One pen tablet. The data underwent a standard EEG pre-processing pipeline, including epoching, bandpass filtering (0.1-60Hz), baseline removal, and artifact removal using Independent Component Analysis (ICA). For signature data, features were extracted by calculating statistical and dynamic measures from raw pen data. Various machine learning models (Gaussian NB, KNN, Linear SVM, Random Forest, Gradient Boost, XGBoost) were then trained and evaluated on unimodal (EEG or Wacom) and multimodal (EEG + Wacom) feature sets for both identification and verification tasks.

Datasets

The paper introduces the SignEEG v1.0 dataset, a new benchmark for multimodal biometric research. This dataset comprises:

  • 70 healthy participants.
  • EEG data collected using an Emotiv Insight headset.
  • Handwritten signature data captured on a Wacom One pen tablet.
  • Data collected across three experimental paradigms: mental imagery of signature image, mental imagery of signature trajectory, and physical signature execution.

The dataset is available in EDF (EEG), CSV (signature), and MATLAB (.mat) formats.

Key Insights and Ideas

  • Multimodality is Key: Combining physiological (EEG) and behavioral (signature) modalities significantly boosts biometric system robustness and accuracy, particularly against forgery.
  • EEG's Forgery Resistance: EEG signals are inherently difficult to forge, making them a strong complement to signature-based biometrics.
  • Diverse Paradigms Matter: The three distinct experimental paradigms (visualizing, mentally simulating, and physically drawing) capture different aspects of cognitive and motor processes related to signature generation, providing a richer dataset.
  • XGBoost and Random Forest Shine: These models consistently perform well across various feature sets and tasks, especially when combining EEG and signature data.
  • Data Availability is Crucial: The release of the SignEEG v1.0 dataset is a significant contribution, enabling further research and development in this domain.

Key Results

The study demonstrates that multimodal approaches significantly outperform unimodal ones. For identification, combining statistical EEG features with Wacom signature features yielded the highest accuracy, with XGBoost achieving 98.68% and Random Forest reaching 97.63%. For verification, similar trends were observed, with Random Forest achieving 98.94% when using statistical, wavelet, and Wacom features combined. Even with limited sample sizes, the multimodal approach achieved high reliability.

Limitations and Future Work

The authors acknowledge that the longitudinal temporal dynamics of EEG have not been explored. Future work will focus on extending data collection to capture these temporal dynamics over prolonged periods to understand how biometric modalities evolve and interact over time, aiming to further improve system accuracy and reliability. Additionally, while the dataset is comprehensive, the authors note that 5 subjects' data was set aside for future robustness analysis, suggesting potential for further exploration with this subset.

Summary updated

Abstract

ABSTRACT Handwritten signatures in biometric authentication leverage unique individual characteristics for identification, offering high specificity through dynamic and static properties. However, this modality faces significant challenges from sophisticated forgery attempts, underscoring the need for enhanced security measures in common applications. To address forgery in signature-based biometric systems, integrating a forgery-resistant modality, namely, noninvasive electroencephalography (EEG), which captures unique brain activity patterns, can significantly enhance system robustness by leveraging multimodality’s strengths. By combining EEG, a physiological modality, with handwritten signatures, a behavioral modality, our approach capitalizes on the strengths of both, significantly fortifying the robustness of biometric systems through this multimodal integration. In addition, EEG’s resistance to replication offers a high-security level, making it a robust addition to user identification and verification. This study presents a new multimodal SignEEG v1.0 dataset based on EEG and hand-drawn signatures from 70 subjects. EEG signals and hand-drawn signatures have been collected with Emotiv Insight and Wacom One sensors, respectively. The multimodal data consists of three paradigms based on mental, & motor imagery, and physical execution: i) thinking of the signaturés image, (ii) drawing the signature mentally, and (iii) drawing a signature physically. Extensive experiments have been conducted to establish a baseline with machine learning classifiers. The results demonstrate that multimodality in biometric systems significantly enhances robustness, achieving high reliability even with limited sample sizes. We release the raw, pre-processed data and easy-to-follow implementation details.
BibTeX
@inproceedings{mishra2024signeeg,
  author = {Ashish Ranjan Mishra and Rakesh Kumar and Vibha Gupta and Sameer M. Prabhu and Richa Upadhyay and Prakash Chandra Chhipa and Sumit Rakesh and Hamam Mokayed and Debashis Das Chakladar and Kanjar De and Marcus Liwicki and Foteini Liwicki and Rajkumar Saini},
  title = {{SignEEG v1.0 : Multimodal Dataset with Electroencephalography and Hand-written Signature for Biometric Systems}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41597-024-03546-z},
}
Figure