TL;DR
This paper proposes a specialized Multi-stream Neural Network (MSNN) for word-level sign language recognition (WSLR) that moves beyond general action recognition. By combining global video context with magnified local views of hands and faces and skeletal joint data, the authors achieved a 10%–15% improvement in Top-1 accuracy over standard baselines on major American Sign Language datasets.
What Problem Does This Paper Address?
While sign language is technically a sequence of actions, the authors argue that Word-level Sign Language Recognition (WSLR) is fundamentally different from general action recognition (like distinguishing "running" from "walking"). WSLR is a fine-grained task. Many signs share similar global body movements but are distinguished only by subtle handshapes, facial expressions, or the precise position of a hand relative to the face.
Existing state-of-the-art methods often rely on models like I3D, which are excellent at capturing spatiotemporal features from whole video frames but often miss these critical, localized details. Furthermore, background noise and variations in a signer's appearance can distract these global models. The challenge is to build a system that can "zoom in" on the informative parts of a sign while maintaining a sense of the overall motion.
Methods
The authors introduce a Multi-stream Neural Network (MSNN) that processes three distinct types of information in parallel:
- Base Stream: This follows the conventional approach, using two sub-streams (one for RGB video frames and one for optical flow) processed by I3D networks to capture global appearance and motion.
- Local Image Stream: This is the "zoom" feature. Using OpenPose to detect keypoints, the system crops and magnifies three regions: the left hand, the right hand, and the face. Each crop is fed into its own I3D network to capture detailed handshapes and facial expressions.
- Skeleton Stream: This stream uses a Spatio-Temporal Graph Convolutional Network (ST-GCN) to process 27 skeletal keypoints (5 for the body, 11 for each hand). This allows the model to focus purely on the relative positions and movements of the body parts, effectively filtering out background noise and appearance variations.
The final prediction is made by averaging the classification scores from all sub-streams (late fusion).
Datasets
The model was evaluated on two large-scale American Sign Language (ASL) datasets:
- WLASL (Word-Level American Sign Language): The largest available WSLR dataset, featuring subsets of 100, 300, 1000, and 2000 word classes.
- MS-ASL: A dataset captured from diverse web videos with varying viewpoints and signers. The authors noted that some original video links were broken, so their results on this dataset (using ~75-80% of the original data) serve as an indicative benchmark.
Key Insights and Ideas
- WSLR as Fine-Grained Action Recognition: The authors successfully demonstrate that treating signs like "jabs vs. hooks" in boxing—where the difference is in the detail—is more effective than treating them as entirely different categories of movement.
- The Power of Local Context: A key "aha" moment was the word "man" vs. "woman." Both involve similar hand-to-face movements, but "man" starts at the forehead while "woman" starts at the chin. Global models often confuse these, but the local and skeletal streams provide the necessary precision to distinguish them.
- Skeleton as a Noise Filter: By using ST-GCNs, the model learns the "grammar" of the movement independently of what the signer is wearing or what is happening in the background.
Key Results
The proposed MSNN (specifically "Ours6," which uses all streams) significantly outperformed the I3D baselines:
- WLASL100: Achieved 81.38% Top-1 accuracy, a massive jump from the 65.89% baseline.
- WLASL2000: Even on the most challenging 2000-word subset, accuracy improved from 32.48% to 47.26%.
- MS-ASL: Showed similar trends, with improvements of roughly 10% across most subsets.
- Qualitative Success: The model achieved 100% accuracy on the word "man" in the WLASL100 test set, whereas the baseline model failed to recognize it entirely (0% accuracy), often confusing it with "woman" or "full."
Limitations and Future Work
- Computational Complexity: Combining multiple I3D streams and an ST-GCN creates a very large network. While parallelizable, it requires significant hardware resources, making real-time deployment on mobile devices a challenge.
- Viewpoint Sensitivity: The local cropping method relies on detecting the elbow to find the hand. In close-up shots or extreme side views (common in MS-ASL), these keypoints are often lost, causing the local stream to fail.
- Data Scarcity: As the number of vocabulary words increases, the number of training videos per word typically decreases, leading to a drop in performance.
- Future Directions: The authors suggest exploring model compression (pruning/distillation) for efficiency and developing more robust hand-tracking that doesn't rely on the elbow joint. They also aim to extend this work to continuous sign language recognition (sentences rather than isolated words).
Summary updated