Hand Pose Guided 3D Pooling for Word-level Sign Language Recognition

Al Amin Hosain, Panneer Selvam Santhalingam, Parth H. Pathak, Huzefa Rangwala, Jana KoseckáWACV 2021 · OpenReview

Abstract

Gestures in American Sign Language (ASL) are characterized by fast, highly articulate motion of upper body, including arm movements with complex hand shapes and facial expressions. In this work, we propose a new method for word-level sign recognition from American Sign Language (ASL) using video. Our method uses both motion and hand shape cues while being robust to variations of execution. We exploit the knowledge of the body pose, estimated from an off-the-shelf pose estimator. Using the pose as a guide, we pool spatio-temporal feature maps from different layers of a 3D convolutional neural network. We train separate classifiers using pose guided pooled features from different resolutions and fuse their prediction scores during test time. This leads to a significant improvement in performance on the WLASL benchmark dataset [25]. The proposed approach achieves 10%, 12%, 9.5% and 6.5% performance gain on WLASL100, WLASL300, WLASL1000, WLASL2000 subsets respectively. To demonstrate the robustness of the pose guided pooling and proposed fusion mechanism, we also evaluate our method by fine tuning the model on another dataset. This yields 10% performance improvement for the proposed method using only 0.4% training data during fine tuning stage.

Introduction

Sign language recognition (SLR) is a uniquely challenging computer vision task. Unlike general action recognition, signs are defined by fast, highly articulated movements where the semantic difference between two words often hinges on subtle changes in hand shape or specific finger orientations.

Sign language gestures involve fast, articulate movements of the hands and upper body.

Standard 3D Convolutional Neural Networks (CNNs), such as the Inflated 3D ConvNet (I3D), excel at capturing global spatio-temporal patterns. However, as data flows through the network, fine-grained spatial details of the hands often become blurred. To address this, we introduce a Pose Guided Pooling (PPC) mechanism that directs the network's attention to the hands using spatial priors from an off-the-shelf pose estimator.

Methodology

The proposed architecture is a multi-stream fusion model built upon an I3D backbone. The core innovation is the PPC module, which operates in four stages:

  1. Pose Estimation: OpenPose is used to detect 2D joint locations (specifically wrists and hands) for every video frame.
  2. Feature Mapping: These (x, y) coordinates are scaled to match the spatial dimensions of intermediate feature maps within the I3D backbone (specifically at Inception blocks 5, 7, and 8).
  3. Localized Pooling: Instead of global pooling, the PPC module extracts a small 3x3x3 volume centered on the estimated hand locations. This captures the "hand shape" features that global models often overlook.
  4. Multi-Scale Fusion: Separate linear classifiers are trained for these localized features (PPC-5, PPC-7, PPC-8). During inference, their prediction scores are fused with the standard global I3D head.
The PPC module extracts localized features from intermediate 3D CNN layers using pose-estimated hand coordinates.

Experimental Results

The method was evaluated on the WLASL (Word-Level American Sign Language) dataset, currently the largest public benchmark for this task. It features "in-the-wild" videos with significant variations in lighting, background, and signing style.

Performance on WLASL

The proposed method consistently outperformed the state-of-the-art I3D baseline across all WLASL subsets:

  • WLASL100: 75.67% Top-1 accuracy (+10% gain over baseline).
  • WLASL300: 68.30% Top-1 accuracy (+12% gain over baseline).
  • WLASL1000: (+9.5% gain over baseline).
  • WLASL2000: 38.84% Top-1 accuracy (+6.5% gain over baseline).

Transferability and Data Efficiency

The approach demonstrated remarkable robustness in transfer learning. By using pose-guided features, the model achieved a 10% improvement over the baseline when fine-tuning on a new dataset using only 0.4% of the available training data. This suggests that the spatial prior provided by pose estimation significantly reduces the data required for the model to learn relevant features.

Discussion and Insights

Analysis shows that the global I3D model and the PPC branches have complementary strengths. The global model is better at signs with heavy motion, while the PPC branches (particularly from intermediate layers like PPC-7) excel at signs with subtle hand shapes and less overall body movement. Using pose as a "shortcut" provides a spatial prior that avoids the need for the CNN to learn hand localization from scratch.

However, the system's performance is dependent on the accuracy of the pose estimator. If the estimator fails due to extreme motion blur or occlusion, the PPC module cannot extract meaningful features.

Conclusion

This work demonstrates that guiding 3D CNNs with pose information significantly improves word-level sign language recognition. Future work will explore extending this mechanism to phrase-level recognition and continuous video streams.

Summary updated

Abstract

Gestures in American Sign Language (ASL) are characterized by fast, highly articulate motion of upper body, including arm movements with complex hand shapes and facial expressions. In this work, we propose a new method for word-level sign recognition from American Sign Language (ASL) using video. Our method uses both motion and hand shape cues while being robust to variations of execution. We exploit the knowledge of the body pose, estimated from an off-the-shelf pose estimator. Using the pose as a guide, we pool spatio-temporal feature maps from different layers of a 3D convolutional neural network. We train separate classifiers using pose guided pooled features from different resolutions and fuse their prediction scores during test time. This leads to a significant improvement in performance on the WLASL benchmark dataset [25]. The proposed approach achieves 10%, 12%, 9.5% and 6.5% performance gain on WLASL100, WLASL300, WLASL1000, WLASL2000 subsets respectively. To demonstrate the robustness of the pose guided pooling and proposed fusion mechanism, we also evaluate our method by fine tuning the model on another dataset. This yields 10% performance improvement for the proposed method using only 0.4% training data during fine tuning stage.
BibTeX
@inproceedings{hosain2021hand,
  author = {Al Amin Hosain and Panneer Selvam Santhalingam and Parth H. Pathak and Huzefa Rangwala and Jana Kosecká},
  title = {{Hand Pose Guided 3D Pooling for Word-level Sign Language Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=7It3TH4Q-H},
}
Figure