3DAirSig: A Framework for Enabling In-Air Signatures Using a Multi-Modal Depth Sensor

Jameel Malik, Ahmed Elhayek, Sheraz Ahmed, Faisal Shafait, Muhammad Imran Malik, Didier StrickerSensors 2018 · OpenReview

TL;DR

This paper introduces 3DAirSig, a novel framework for in-air signature verification that leverages the often-overlooked depth dimension captured by depth sensors. By employing a CNN-based hand pose estimation technique to accurately track 3D fingertip trajectories and introducing a new dataset, 3DAirSig demonstrates that depth information alone is a powerful and sufficient feature for robust user authentication.

What Problem Does This Paper Address?

Traditional signature verification methods, whether offline (on paper) or online (on touchscreens), primarily rely on 2D spatial and temporal information. In-air signatures, a more recent modality allowing users to sign in the air, offer a richer 3D representation. However, existing approaches largely treat these 3D signatures as 2D trajectories, failing to exploit the significant potential of the depth dimension. This oversight, coupled with unreliable heuristic methods for tracking the user's fingertip, limits the accuracy and robustness of in-air signature verification systems. The paper aims to bridge this gap by developing a system that effectively captures and utilizes the 3D depth information for more secure and reliable authentication.

Methods

The 3DAirSig framework consists of two main phases: acquisition and verification.

Acquisition: * CNN-based Hand Pose Estimation: Instead of relying on fragile heuristic fingertip detection, the system uses a Convolutional Neural Network (CNN) called PoseCNN. This network takes a single depth image as input and estimates the 3D positions of 16 hand joints. This approach is more robust to occlusions and variations in hand pose. * Fingertip Tracking: The 3D position of the index fingertip is extracted from the estimated hand pose for each frame, forming the 3D signature trajectory. * Data Acquisition Setup: The system utilizes an Intel Creative Senz3D depth camera with a 74° diagonal field of view. Position markers are used to define the signing area, and users can choose between left or right hand. A real-time 2D projection of the 3D trajectory is displayed on a monitor for user feedback.

Verification: * Preprocessing: Signatures undergo normalization and smoothing to remove noise and compensate for variations in scale and position. * Feature Extraction: The paper explores several feature combinations: * 2D Spatial (X,Y) * Depth (Z) * Spatial (X,Z) * Spatial (Y,Z) * Combined Spatial+Depth (X,Y,Z) * Matching: The Multidimensional Dynamic Time Warping (MD-DTW) algorithm is employed for matching the extracted features against precomputed templates. MD-DTW is chosen for its ability to align temporal signals effectively, even with inconsistencies. * Decision Making: A decision threshold is used to classify a signature as genuine or forged based on the similarity score obtained from MD-DTW.

Datasets

The paper introduces a newly created dataset for in-air signature verification.

  • 3DAirSig Dataset: This dataset comprises 600 signatures from 15 different subjects. For each subject, 15 genuine signatures were recorded, and 25 forgeries were obtained from 5 impostors. Ten genuine signatures per user were used for testing, and the remaining five for training. The dataset is publicly available.

Key Insights and Ideas

  • Depth is a Powerful Feature: The most significant insight is that the 1D depth (Z) component of an in-air signature trajectory, when analyzed independently, provides a surprisingly strong and sufficient basis for verification. This challenges the conventional focus on 2D spatial information.
  • Depth is Hard to Forge: Even when impostors were shown video recordings of genuine signatures from multiple angles, they struggled to replicate the unique depth patterns, highlighting its inherent security advantage.
  • CNNs for Robust Hand Pose: Leveraging deep learning for full 3D hand pose estimation, rather than heuristic fingertip tracking, significantly improves the stability and reliability of signature acquisition, especially in the presence of occlusions.
  • Multi-modal Fusion Potential: While depth alone is effective, combining it with spatial information (3D-SV module) further enhances performance, suggesting that a multi-modal approach offers the best of both worlds.

Key Results

  • Overall Performance: The proposed 3DAirSig framework, specifically the 3D-SV module (utilizing full 3D information), achieved an Equal Error Rate (EER) of 0.46%.
  • Depth-Only Performance: The Depth-based Signature Verification (DSV) module, using only the 1D depth feature, achieved an EER of 0.51%, demonstrating its strong standalone performance.
  • Comparison of Modules:
    • DSV (Depth only): EER 0.51%
    • SSV (2D Spatial only): EER 0.69%
    • ISSV (Improved 2D Spatial): EER 0.58%
    • 3D-SV (Full 3D): EER 0.46%
  • Dataset Contribution: The creation of a new, publicly available dataset is a significant contribution to the field, enabling further research and benchmarking.

Limitations and Future Work

  • Limitations:
    • The current system relies on a specific depth camera (Intel Creative Senz3D).
  • While the CNN approach is robust, the performance is still dependent on the accuracy of the hand pose estimation.
  • The paper notes that the ISSV module, while improved over SSV, still lagged behind the DSV module, suggesting that simply combining spatial features might not be as effective as leveraging depth directly or fully 3D information.
  • Future Work:
    • The authors plan to extend their dataset with more signatures and subjects.
  • They aim to develop a CNN-based algorithm specifically for in-air signature classification and matching, potentially moving beyond MD-DTW for improved efficiency or accuracy.
  • Further exploration of multi-modal fusion techniques could lead to even more robust systems.

Summary updated

Abstract

In-air signature is a new modality which is essential for user authentication and access control in noncontact mode and has been actively studied in recent years. However, it has been treated as a conventional online signature, which is essentially a 2D spatial representation. Notably, this modality bears a lot more potential due to an important hidden depth feature. Existing methods for in-air signature verification neither capture this unique depth feature explicitly nor fully explore its potential in verification. Moreover, these methods are based on heuristic approaches for fingertip or hand palm center detection, which are not feasible in practice. Inspired by the great progress in deep-learning-based hand pose estimation, we propose a real-time in-air signature acquisition method which estimates hand joint positions in 3D using a single depth image. The predicted 3D position of fingertip is recorded for each frame. We present four different implementations of a verification module, which are based on the extracted depth and spatial features. An ablation study was performed to explore the impact of the depth feature in particular. For matching, we employed the most commonly used multidimensional dynamic time warping (MD-DTW) algorithm. We created a new database which contains 600 signatures recorded from 15 different subjects. Extensive evaluations were performed on our database. Our method, called 3DAirSig, achieved an equal error rate (EER) of 0.46 %. Experiments showed that depth itself is an important feature, which is sufficient for in-air signature verification.
BibTeX
@inproceedings{malik2018dairsig,
  author = {Jameel Malik and Ahmed Elhayek and Sheraz Ahmed and Faisal Shafait and Muhammad Imran Malik and Didier Stricker},
  title = {{3DAirSig: A Framework for Enabling In-Air Signatures Using a Multi-Modal Depth Sensor}},
  booktitle = {Sensors},
  year = {2018},
  url = {https://openreview.net/forum?id=b8wjJ7XPmi},
}
Figure