A Non-Anatomical Graph Structure for isolated hand gesture separation in continuous gesture sequences

Razieh Rastgoo, Kourosh Kiani, Sergio EscaleraarXiv 2022 · DOI · arXiv

Recognizing isolated hand gestures is a well-established task in computer vision. However, transitioning to continuous gesture recognition—such as interpreting full sentences in sign language—presents a significant challenge. The primary difficulty lies in "boundary detection": identifying precisely when one gesture ends and the next begins within a seamless stream of motion.

A recent study by Rastgoo et al. introduces a novel approach to this problem by utilizing Graph Convolutional Networks (GCNs) with "non-anatomical" graph structures. By deviating from traditional biological bone structures, the proposed model achieves superior performance in segmenting and identifying gestures in continuous sequences.

Methodology

The authors propose an end-to-end multi-stage pipeline designed to automate feature extraction and improve robustness against noisy backgrounds and varying camera angles.

The proposed multi-stage architecture for continuous hand gesture recognition, from detection to classification.

The pipeline consists of the following stages:

  1. Detection & Pose Estimation: A Single Shot Detector (SSD) identifies the hand within the video frames. Subsequently, a 3D pose estimator (based on OpenPose) extracts 21 keypoints representing the hand joints.
  2. Spatial Feature Extraction via GCN: Rather than treating joint coordinates as a simple vector, the model interprets them as nodes in a graph. A two-layer GCN is employed to learn spatial relationships between these joints.
  3. Temporal Modeling: The spatial embeddings from the GCN are passed into a stacked Bidirectional Long Short-Term Memory (Bi-LSTM) network coupled with an Attention mechanism. This stage captures the temporal "flow" and focuses on the most informative frames of a gesture.
  4. Post-Processing: A sliding window approach and softmax thresholding are used to determine the final gesture boundaries and class labels.

Innovation: Non-Anatomical Graph Topologies

The core innovation of this research lies in the exploration of different graph topologies for the GCN. The authors compared three distinct structures to determine which best captures the nuances of hand movement for boundary detection.

Three graph topologies evaluated: (a) Anatomical, (b) Finger-reference, and (c) Palm-reference (Non-Anatomical).
  • Anatomical (Type A): Joints are connected according to the actual skeletal structure of the human hand.
  • Finger-Reference (Type B): Connections are established starting from specific points on the fingers and the palm.
  • Palm-Reference (Type C): This non-anatomical structure connects every joint directly to the palm keypoint. In this configuration, the palm acts as a central "gravity center" or reference point for all other movements.

The study found that the Palm-Reference (Type C) graph significantly outperformed the others. The authors hypothesize that because the palm moves more smoothly and consistently than the fingers during a gesture, it provides a stable reference frame that allows the network to more accurately calculate the relative motion of the digits.

Experimental Results

The model was evaluated on two major benchmarks: the RKS-PERSIANSIGN dataset (10,000 videos of Persian gestures) and the ASLVID (American Sign Language Lexicon Video Dataset). To simulate continuous sequences, isolated gestures were concatenated, a standard practice in the field due to the scarcity of large-scale annotated continuous datasets.

The results indicate a substantial improvement over previous state-of-the-art methods:

  • Accuracy: The non-anatomical model achieved 99.90% accuracy on the RKS-PERSIANSIGN dataset and 96.00% on the ASLVID dataset.
  • Error Reduction: False recognitions (misidentified boundaries or phantom gestures) were reduced by over 60% on the RKS dataset compared to previous handcrafted feature-based methods.
  • Efficiency: By replacing computationally expensive handcrafted features (like SVD-based descriptors) with automated GCN embeddings, the system is better suited for real-time applications.

Conclusion and Future Work

The success of the non-anatomical graph structure suggests that for neural networks, biological accuracy is not always the optimal path for feature representation. By treating the palm as a stable anchor, the GCN can more effectively learn the spatial-temporal patterns necessary for precise gesture segmentation.

While highly effective on simulated continuous data, future research will focus on "natural" continuous sign language, which involves complex co-articulation where signs blend into one another. Additionally, the authors aim to apply these non-anatomical graph principles to full-body Human Action Recognition (HAR) to see if similar "gravity centers" exist for broader human movements.

Summary updated

Abstract

Continuous Hand Gesture Recognition (CHGR) has been extensively studied by researchers in the last few decades. Recently, one model has been presented to deal with the challenge of the boundary detection of isolated gestures in a continuous gesture video [17]. To enhance the model performance and also replace the handcrafted feature extractor in the presented model in [17], we propose a GCN model and combine it with the stacked Bi-LSTM and Attention modules to push the temporal information in the video stream. Considering the breakthroughs of GCN models for skeleton modality, we propose a two-layer GCN model to empower the 3D hand skeleton features. Finally, the class probabilities of each isolated gesture are fed to the post-processing module, borrowed from [17]. Furthermore, we replace the anatomical graph structure with some non-anatomical graph structures. Due to the lack of a large dataset, including both the continuous gesture sequences and the corresponding isolated gestures, three public datasets in Dynamic Hand Gesture Recognition (DHGR), RKS-PERSIANSIGN, and ASLVID, are used for evaluation. Experimental results show the superiority of the proposed model in dealing with isolated gesture boundaries detection in continuous gesture sequences
BibTeX
@misc{rastgoo2022non,
  author = {Razieh Rastgoo and Kourosh Kiani and Sergio Escalera},
  title = {{A Non-Anatomical Graph Structure for isolated hand gesture separation in continuous gesture sequences}},
  year = {2022},
  eprint = {2207.07619},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.07619},
}
Figure