Introduction
Sign Language Recognition (SLR) is a complex task due to the high variability in hand shapes, orientations, and body postures. Traditional deep learning models face an "annotation bottleneck," requiring vast amounts of labeled data for every sign. This paper introduces ZS-SLR, a novel framework for Zero-Shot Sign Language Recognition that identifies signs never encountered during training by bridging the gap between visual kinematics and linguistic meaning.
Methodology: The ZS-SLR Framework
The authors propose a sophisticated multi-stage pipeline that moves away from traditional CNN-only approaches, leveraging both visual and semantic information.
Multi-Stream Processing and Detection
The model processes both RGB (color/texture) and Depth (3D structure) videos to capture complementary information. Instead of standard detectors like YOLO, the framework employs a DEtection TRansformer (DETE) for human detection, which is better suited for human-centric video challenges.
Kinematic Segmentation and Feature Extraction
Using OpenPose, the system extracts 13 keypoints and segments the human body into nine distinct parts. This kinematic segmentation allows the model to focus on the relative motion of limbs, which is critical for distinguishing signs that appear similar in a global view.
These nine segments are fed into a Vision Transformer (ViT) to capture spatial features, followed by an LSTM network to learn the temporal "flow" of the sign over time.
Semantic Mapping for Zero-Shot Learning
To enable zero-shot capabilities, the model uses BERT to generate 1024-dimensional embeddings of sign labels. The visual features are projected into this semantic space. At test time, an unseen video is classified by finding the nearest BERT embedding to its predicted visual projection, effectively leveraging linguistic "prior knowledge."
Datasets and Evaluation
The ZS-SLR model was evaluated on four diverse benchmarks:
- Montalbano II: 20 Italian gesture classes.
- MSR Daily Activity 3D: 16 daily human activity classes.
- CAD-60: 12 office and home activity classes.
- NTU-60: A large-scale dataset with 60 classes, used to test performance at scale.
Key Insights and Results
- Multimodal Synergy: The combined RGB-D model significantly outperformed single-stream versions, achieving 62.4% accuracy on Montalbano compared to 50.3% for RGB-only.
- State-of-the-Art Performance: On the NTU-60 dataset, ZS-SLR reached 79.2% accuracy, outperforming previous SOTA models like GZSL and CADA-VAE.
- Cross-Domain Success: Beyond SLR, the model achieved SOTA results for Zero-Shot Gesture Recognition (ZS-GR) and Zero-Shot Action Recognition (ZS-AR).
Conclusion
ZS-SLR demonstrates the power of combining Vision Transformers with semantic embeddings for zero-shot tasks. While the model currently struggles with highly similar signs, such as "shake head" versus "rub hands," future work will focus on refining the semantic space and increasing sample diversity to improve fine-grained discrimination.
Summary updated