Word-level Sign Language Recognition with Multi-stream Neural Networks Focusing on Local Regions and Skeletal Information

Mizuki Maruyama, Shrey Singh, Katsufumi Inoue, Partha Pratim Roy, Masakazu Iwamura, Michifumi YoshiokaarXiv 2021 · DOI · arXiv

TL;DR

This paper proposes a specialized Multi-stream Neural Network (MSNN) for word-level sign language recognition (WSLR) that moves beyond general action recognition. By combining global video context with magnified local views of hands and faces and skeletal joint data, the authors achieved a 10%–15% improvement in Top-1 accuracy over standard baselines on major American Sign Language datasets.

What Problem Does This Paper Address?

While sign language is technically a sequence of actions, the authors argue that Word-level Sign Language Recognition (WSLR) is fundamentally different from general action recognition (like distinguishing "running" from "walking"). WSLR is a fine-grained task. Many signs share similar global body movements but are distinguished only by subtle handshapes, facial expressions, or the precise position of a hand relative to the face.

Existing state-of-the-art methods often rely on models like I3D, which are excellent at capturing spatiotemporal features from whole video frames but often miss these critical, localized details. Furthermore, background noise and variations in a signer's appearance can distract these global models. The challenge is to build a system that can "zoom in" on the informative parts of a sign while maintaining a sense of the overall motion.

Methods

The authors introduce a Multi-stream Neural Network (MSNN) that processes three distinct types of information in parallel:

  1. Base Stream: This follows the conventional approach, using two sub-streams (one for RGB video frames and one for optical flow) processed by I3D networks to capture global appearance and motion.
  2. Local Image Stream: This is the "zoom" feature. Using OpenPose to detect keypoints, the system crops and magnifies three regions: the left hand, the right hand, and the face. Each crop is fed into its own I3D network to capture detailed handshapes and facial expressions.
  3. Skeleton Stream: This stream uses a Spatio-Temporal Graph Convolutional Network (ST-GCN) to process 27 skeletal keypoints (5 for the body, 11 for each hand). This allows the model to focus purely on the relative positions and movements of the body parts, effectively filtering out background noise and appearance variations.
FIGURE 1: Overview of the proposed method. The pro- posed multi-stream neural network (MSNN) consists of three streams: 1) a base stream, 2) local image stream, and 3) skeleton stream. Each stream is trained separately, and the recognition scores extracted from each stream are averaged to obtain the final recognition result.

The final prediction is made by averaging the classification scores from all sub-streams (late fusion).

Datasets

The model was evaluated on two large-scale American Sign Language (ASL) datasets:

  • WLASL (Word-Level American Sign Language): The largest available WSLR dataset, featuring subsets of 100, 300, 1000, and 2000 word classes.
  • MS-ASL: A dataset captured from diverse web videos with varying viewpoints and signers. The authors noted that some original video links were broken, so their results on this dataset (using ~75-80% of the original data) serve as an indicative benchmark.

Key Insights and Ideas

  • WSLR as Fine-Grained Action Recognition: The authors successfully demonstrate that treating signs like "jabs vs. hooks" in boxing—where the difference is in the detail—is more effective than treating them as entirely different categories of movement.
  • The Power of Local Context: A key "aha" moment was the word "man" vs. "woman." Both involve similar hand-to-face movements, but "man" starts at the forehead while "woman" starts at the chin. Global models often confuse these, but the local and skeletal streams provide the necessary precision to distinguish them.
  • Skeleton as a Noise Filter: By using ST-GCNs, the model learns the "grammar" of the movement independently of what the signer is wearing or what is happening in the background.
FIGURE 5: Overview of the ST-GCN architecture [9].

Key Results

The proposed MSNN (specifically "Ours6," which uses all streams) significantly outperformed the I3D baselines:

  • WLASL100: Achieved 81.38% Top-1 accuracy, a massive jump from the 65.89% baseline.
  • WLASL2000: Even on the most challenging 2000-word subset, accuracy improved from 32.48% to 47.26%.
  • MS-ASL: Showed similar trends, with improvements of roughly 10% across most subsets.
  • Qualitative Success: The model achieved 100% accuracy on the word "man" in the WLASL100 test set, whereas the baseline model failed to recognize it entirely (0% accuracy), often confusing it with "woman" or "full."

Limitations and Future Work

  • Computational Complexity: Combining multiple I3D streams and an ST-GCN creates a very large network. While parallelizable, it requires significant hardware resources, making real-time deployment on mobile devices a challenge.
  • Viewpoint Sensitivity: The local cropping method relies on detecting the elbow to find the hand. In close-up shots or extreme side views (common in MS-ASL), these keypoints are often lost, causing the local stream to fail.
  • Data Scarcity: As the number of vocabulary words increases, the number of training videos per word typically decreases, leading to a drop in performance.
  • Future Directions: The authors suggest exploring model compression (pruning/distillation) for efficiency and developing more robust hand-tracking that doesn't rely on the elbow joint. They also aim to extend this work to continuous sign language recognition (sentences rather than isolated words).

Summary updated

Abstract

Word-level sign language recognition (WSLR) has attracted attention because it is expected to overcome the communication barrier between people with speech impairment and those who can hear. In the WSLR problem, a method designed for action recognition has achieved the state-of-the-art accuracy. Indeed, it sounds reasonable for an action recognition method to perform well on WSLR because sign language is regarded as an action. However, a careful evaluation of the tasks reveals that the tasks of action recognition and WSLR are inherently different. Hence, in this paper, we propose a novel WSLR method that takes into account information specifically useful for the WSLR problem. We realize it as a multi-stream neural network (MSNN), which consist of three streams: 1) base stream, 2) local image stream, and 3) skeleton stream. Each stream is designed to handle different types of information. The base stream deals with quick and detailed movements of the hands and body, the local image stream focuses on handshapes and facial expressions, and the skeleton stream captures the relative positions of the body and both hands. This approach allows us to combine various types of data for more comprehensive gesture analysis. Experimental results on the WLASL and MS-ASL datasets show the effectiveness of the proposed method; it achieved an improvement of approximately 10\%--15\% in Top-1 accuracy when compared with conventional methods.
BibTeX
@misc{maruyama2021word,
  author = {Mizuki Maruyama and Shrey Singh and Katsufumi Inoue and Partha Pratim Roy and Masakazu Iwamura and Michifumi Yoshioka},
  title = {{Word-level Sign Language Recognition with Multi-stream Neural Networks Focusing on Local Regions and Skeletal Information}},
  year = {2021},
  eprint = {2106.15989},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.15989},
}

Loading related content...

Figure