ZS-SLR: Zero-Shot Sign Language Recognition from RGB-D Videos

Razieh Rastgoo, Kourosh Kiani, Sergio EscaleraarXiv 2021 · arXiv

Introduction

Sign Language Recognition (SLR) is a complex task due to the high variability in hand shapes, orientations, and body postures. Traditional deep learning models face an "annotation bottleneck," requiring vast amounts of labeled data for every sign. This paper introduces ZS-SLR, a novel framework for Zero-Shot Sign Language Recognition that identifies signs never encountered during training by bridging the gap between visual kinematics and linguistic meaning.

Methodology: The ZS-SLR Framework

The authors propose a sophisticated multi-stage pipeline that moves away from traditional CNN-only approaches, leveraging both visual and semantic information.

The overall architecture of the ZS-SLR framework, illustrating the two-stream processing of RGB and Depth videos through detection, segmentation, and classification.

Multi-Stream Processing and Detection

The model processes both RGB (color/texture) and Depth (3D structure) videos to capture complementary information. Instead of standard detectors like YOLO, the framework employs a DEtection TRansformer (DETE) for human detection, which is better suited for human-centric video challenges.

Kinematic Segmentation and Feature Extraction

Using OpenPose, the system extracts 13 keypoints and segments the human body into nine distinct parts. This kinematic segmentation allows the model to focus on the relative motion of limbs, which is critical for distinguishing signs that appear similar in a global view.

Detailed view of the Vision Transformer stage, showing how segmented body parts are processed as patches to capture spatial features.

These nine segments are fed into a Vision Transformer (ViT) to capture spatial features, followed by an LSTM network to learn the temporal "flow" of the sign over time.

Semantic Mapping for Zero-Shot Learning

To enable zero-shot capabilities, the model uses BERT to generate 1024-dimensional embeddings of sign labels. The visual features are projected into this semantic space. At test time, an unseen video is classified by finding the nearest BERT embedding to its predicted visual projection, effectively leveraging linguistic "prior knowledge."

Datasets and Evaluation

The ZS-SLR model was evaluated on four diverse benchmarks:

  • Montalbano II: 20 Italian gesture classes.
  • MSR Daily Activity 3D: 16 daily human activity classes.
  • CAD-60: 12 office and home activity classes.
  • NTU-60: A large-scale dataset with 60 classes, used to test performance at scale.

Key Insights and Results

  • Multimodal Synergy: The combined RGB-D model significantly outperformed single-stream versions, achieving 62.4% accuracy on Montalbano compared to 50.3% for RGB-only.
  • State-of-the-Art Performance: On the NTU-60 dataset, ZS-SLR reached 79.2% accuracy, outperforming previous SOTA models like GZSL and CADA-VAE.
  • Cross-Domain Success: Beyond SLR, the model achieved SOTA results for Zero-Shot Gesture Recognition (ZS-GR) and Zero-Shot Action Recognition (ZS-AR).

Conclusion

ZS-SLR demonstrates the power of combining Vision Transformers with semantic embeddings for zero-shot tasks. While the model currently struggles with highly similar signs, such as "shake head" versus "rub hands," future work will focus on refining the semantic space and increasing sample diversity to improve fine-grained discrimination.

Summary updated

Abstract

Sign Language Recognition (SLR) is a challenging research area in computer vision. To tackle the annotation bottleneck in SLR, we formulate the problem of Zero-Shot Sign Language Recognition (ZS-SLR) and propose a two-stream model from two input modalities: RGB and Depth videos. To benefit from the vision Transformer capabilities, we use two vision Transformer models, for human detection and visual features representation. We configure a transformer encoder-decoder architecture, as a fast and accurate human detection model, to overcome the challenges of the current human detection models. Considering the human keypoints, the detected human body is segmented into nine parts. A spatio-temporal representation from human body is obtained using a vision Transformer and a LSTM network. A semantic space maps the visual features to the lingual embedding of the class labels via a Bidirectional Encoder Representations from Transformers (BERT) model. We evaluated the proposed model on four datasets, Montalbano II, MSR Daily Activity 3D, CAD-60, and NTU-60, obtaining state-of-the-art results compared to state-of-the-art ZS-SLR models.
BibTeX
@misc{rastgoo2021zs,
  author = {Razieh Rastgoo and Kourosh Kiani and Sergio Escalera},
  title = {{ZS-SLR: Zero-Shot Sign Language Recognition from RGB-D Videos}},
  year = {2021},
  eprint = {2108.10059},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.10059},
}
Figure