FineHand: Learning Hand Shapes for American Sign Language Recognition

Al Amin Hosain, Panneer Selvam Santhalingam, Parth H. Pathak, Huzefa Rangwala, Jana Kosecka

Abstract

American Sign Language (ASL) recognition is a difficult gesture recognition problem, characterized by fast, highly articulate gestures. These are comprised of arm movements with different hand shapes, facial expression and head movements. Among these components, hand shape is the vital, often the most discriminative part of a gesture. In this work, we present an approach for effective learning of hand shape embeddings, which are discriminative for ASL gestures. For hand shape recognition our method uses a mix of manually labelled hand shapes and high confidence predictions to train deep convolutional neural network (CNN). The sequential gesture component is captured by recursive neural network (RNN) trained on the embeddings learned in the first stage. We will demonstrate that higher quality hand shape models can significantly improve the accuracy of final video gesture classification in challenging conditions with variety of speakers, different illumination and significant motion blur. We compare our model to alternative approaches exploiting different modalities and representations of the data and show improved video gesture recognition accuracy on GMU-ASL51 benchmark dataset.

Introduction

Recognizing American Sign Language (ASL) "in the wild" is notoriously difficult. Signers move quickly, causing significant motion blur; lighting conditions vary; and different signers have unique styles. While many previous approaches attempted to model the whole body or used depth sensors (like the Microsoft Kinect), these methods often miss the most critical piece of information: the specific shape of the hand.

FineHand is a two-stage deep learning framework designed to bridge this gap. By focusing on fine-grained hand shape embeddings rather than coarse body pose data, the system achieves state-of-the-art performance using standard RGB video. The core innovation lies in an iterative, semi-supervised learning process that creates high-quality hand-shape representations with minimal manual labeling effort.

Methodology

The FineHand pipeline consists of two main phases: Hand Shape Embedding and Sequential Gesture Learning.

  1. Hand Localization: The system uses OpenPose to detect 2D hand joints in RGB video frames, allowing it to crop consistent patches of the left and right hands.
  2. Iterative Hand Shape Learning: Instead of labeling thousands of frames, the authors started with a tiny manually labeled set (one sample per sign per subject). They trained a ResNet-50 on this seed data, used the model to predict shapes in new videos, and then manually corrected only the high-confidence errors. After three iterations, they defined 41 distinct hand-shape classes.
  3. Feature Extraction: The penultimate layer of this trained ResNet serves as a 2048-dimensional "embedding"—a mathematical summary of the hand's shape that is robust to rotation and scaling.
Overview of the FineHand pipeline, showing hand localization and the resulting temporal sequences of hand patches for both hands.
  1. Sequential Modeling: These embeddings are fed into two separate Long Short-Term Memory (LSTM) networks (one for each hand). LSTMs are specialized for processing sequences, allowing the model to understand how hand shapes evolve over time to form a specific sign.
  2. Late Fusion: The outputs of the two LSTMs are combined using the mean of the prediction scores to make the final classification.
Detailed sequences of hand shapes extracted for a specific sign, illustrating the fine-grained features captured by the model across different signers.

Experimental Results

The primary benchmark used is GMU-ASL51, which contains 51 word-level ASL signs performed by 12 different subjects. The authors have also released per-frame hand-shape annotations for this dataset to facilitate future research.

  • State-of-the-Art Performance: FineHand achieved 93% average accuracy across 12 subjects, significantly outperforming 3D CNNs (51%) and the multi-modal FoaNet (15%).
  • Outperforming Depth Sensors: Surprisingly, the RGB-only FineHand model (93%) beat the best Kinect-based FusionLSTM model (88%). This suggests that high-resolution visual texture is more important for sign language than 3D spatial coordinates.
  • The Power of Iteration: Moving from generic ImageNet features to specialized hand-shape features (Iteration 3) improved accuracy from 65% to 93%.
  • Bimanual Importance: Using both hands provided a 3% accuracy boost over using just the dominant hand, highlighting the importance of capturing bimanual signs.

Discussion and Conclusion

The success of FineHand demonstrates that hand shape is the primary discriminative feature for ASL. By explicitly training the model to recognize "garbage" (motion blur) and "rest" positions, the sequential model learns to ignore uninformative frames that usually confuse gesture recognizers.

While FineHand excels at isolated word recognition, continuous sign language parsing remains a challenge. Future work will focus on real-time, continuous translation and exploring fully self-supervised ways to cluster hand shapes to further reduce the need for manual intervention.

Summary updated

Abstract

American Sign Language recognition is a difficult gesture recognition problem, characterized by fast, highly articulate gestures. These are comprised of arm movements with different hand shapes, facial expression and head movements. Among these components, hand shape is the vital, often the most discriminative part of a gesture. In this work, we present an approach for effective learning of hand shape embeddings, which are discriminative for ASL gestures. For hand shape recognition our method uses a mix of manually labelled hand shapes and high confidence predictions to train deep convolutional neural network (CNN). The sequential gesture component is captured by recursive neural network (RNN) trained on the embeddings learned in the first stage. We will demonstrate that higher quality hand shape models can significantly improve the accuracy of final video gesture classification in challenging conditions with variety of speakers, different illumination and significant motion blurr. We compare our model to alternative approaches exploiting different modalities and representations of the data and show improved video gesture recognition accuracy on GMU-ASL51 benchmark dataset
BibTeX
@inproceedings{hosain2020finehand,
  author = {Al Amin Hosain and Panneer Selvam Santhalingam and Parth H. Pathak and Huzefa Rangwala and Jana Kosecka},
  title = {{FineHand: Learning Hand Shapes for American Sign Language Recognition}},
  booktitle = {FG},
  year = {2020},
  url = {https://openreview.net/forum?id=LDudv69exqF},
}
Figure