Multi-Modal Zero-Shot Sign Language Recognition

Razieh Rastgoo, Kourosh Kiani, Sergio Escalera, Mohammad SabokrouarXiv 2021 · OpenReview

TL;DR

This paper introduces a novel multi-modal Zero-Shot Sign Language Recognition (ZS-SLR) system that overcomes the significant annotation bottleneck in sign language recognition. By leveraging textual descriptions and fusing skeleton-based and deep visual features, the model can recognize sign languages it hasn't been explicitly trained on, achieving state-of-the-art results.

What Problem Does This Paper Address?

Sign Language Recognition (SLR) is a challenging task that typically requires vast amounts of annotated video data for training. This annotation bottleneck makes it difficult to develop systems that can recognize a wide range of signs, especially new or less common ones. Traditional SLR models struggle to generalize to unseen sign classes. Zero-Shot Learning (ZSL) offers a promising solution by enabling models to recognize classes they haven't encountered during training, usually by learning a mapping between visual features and semantic descriptions. However, applying ZSL to sign language recognition (ZS-SLR) is still an open area with limited research. This paper aims to address this by proposing a ZS-SLR system that can recognize signs without any visual examples of those specific signs during training, relying instead on textual descriptions of the signs.

Methods

The proposed ZS-SLR model is multi-modal and employs an embedding-based ZSL approach. Here's a breakdown of its key components:

  • Transformer-based Hand Detection: The authors use the DETR (DEtection TRansformer) model for robust and end-to-end hand detection, a crucial first step for sign language analysis. This is highlighted as a novel application of Transformers for this specific task.
  • Hand Pose Estimation: Following hand detection, 3D hand keypoints are estimated using the OpenPose model.
  • Hybrid Feature Extraction: The model extracts two types of visual features:
  • Skeleton-based Features: These include distances between hand joints, angles formed by joint triplets, and singular values derived from Singular Value Decomposition (SVD) of hand keypoints. These handcrafted features are designed to be compact and robust.
  • Deep Features: These are extracted from RGB video frames using a C3D network.
  • Feature Fusion and Dimensionality Reduction: To balance the dimensionality of skeleton-based and deep features, an Auto-Encoder (AE) is applied to the output of an LSTM network that processes the C3D features. The skeleton-based features are also repeated and weighted to match the dimensionality of the deep features before fusion.
  • Lingual Embedding: The Bidirectional Encoder Representations from Transformers (BERT) model is used to generate a 1024-dimensional embedding for the textual descriptions of the sign language classes.
  • Semantic Space and Classification: The fused visual features are mapped into a semantic space using a deep neural network. This projection function is trained on seen classes. During inference, the model predicts the semantic embedding of an unseen sign and then classifies it by finding the closest class embedding in the semantic space, using cosine similarity. A final Softmax layer is applied for classification.

Datasets

The proposed model was evaluated on four large-scale datasets:

  • RKS-PERSIANSIGN
  • First-Person
  • ASLVID
  • isoGD

For the First-Person and isoGD datasets, only the RGB samples were used. The paper also mentions that the work by Bilge et al. [27] extended the ASL dataset to ASL-Text, which includes sign descriptions, suggesting a potential benchmark for ZS-SLR.

Key Insights and Ideas

  • Leveraging Text for Zero-Shot Recognition: The core idea of using textual descriptions of signs to bridge the gap between seen and unseen classes is a powerful strategy for overcoming annotation limitations in SLR.
  • Synergy of Skeleton and Deep Features: Combining the compact, interpretable nature of skeleton-based features with the rich representational power of deep features from RGB videos leads to improved performance. This multi-modal approach captures complementary information.
  • Transformer for Hand Detection: Applying Transformer architectures to hand detection is a novel contribution, demonstrating their effectiveness in this specific computer vision task.
  • Balancing Modalities: The use of an Auto-Encoder to reduce the dimensionality of deep features and balance it with skeleton-based features is a practical approach to effectively fuse diverse feature types.

Key Results

The proposed multi-modal ZS-SLR model achieved state-of-the-art performance across all four evaluated datasets. Specifically, when comparing different feature combinations, the fused "Distances + Angles + SVD + C3D + LSTM + BERT" achieved the highest accuracy (e.g., 74.6% on RKS-PERSIANSIGN under Protocol 1). The paper also shows that the Transformer-based hand detection model outperforms other common methods like Faster-RCNN, SSD, and YOLO. The model's performance is consistently high across different evaluation protocols, indicating its robustness.

Limitations and Future Work

  • Limitations:
    • While the model achieves state-of-the-art results, there is still room for improvement in recognition accuracy.
  • Some signs, like 'Sister', 'Brother', 'Alligator', and 'Sailboat', were identified as challenging to discriminate, suggesting that similarities between signs can still lead to misclassifications.
  • The paper notes that even small variations in handshape or motion can significantly alter the meaning of a sign, making SLR inherently difficult.
  • Future Work:
    • The authors suggest exploring the effectiveness of substituting the current CNN model with a Transformer model for feature extraction, referencing the limitations of CNNs in capturing smaller receptive fields.
  • Investigating how to better handle complex patterns in real-world sign language scenarios, especially with unseen data, is a key direction.
  • Further research into dedicated models for ZS-SLR is deemed indispensable due to the nuanced nature of sign language.

Summary updated

Abstract

Zero-Shot Learning (ZSL) has rapidly advanced in recent years. Towards overcoming the annotation bottleneck in the Sign Language Recognition (SLR), we explore the idea of Zero-Shot Sign Language Recognition (ZS-SLR) with no annotated visual examples, by leveraging their textual descriptions. In this way, we propose a multi-modal Zero-Shot Sign Language Recognition (ZS-SLR) model harnessing from the complementary capabilities of deep features fused with the skeleton-based ones. A Transformer-based model along with a C3D model is used for hand detection and deep features extraction, respectively. To make a trade-off between the dimensionality of the skeletonbased and deep features, we use an Auto-Encoder (AE) on top of the Long Short Term Memory (LSTM) network. Finally, a semantic space is used to map the visual features to the lingual embedding of the class labels, achieved via the Bidirectional Encoder Representations from Transformers (BERT) model. Results on four large-scale datasets, RKS-PERSIANSIGN, First-Person, ASLVID, and isoGD, show the superiority of the proposed model compared to state-of-the-art alternatives in ZS-SLR.
BibTeX
@misc{rastgoo2021multi,
  author = {Razieh Rastgoo and Kourosh Kiani and Sergio Escalera and Mohammad Sabokrou},
  title = {{Multi-Modal Zero-Shot Sign Language Recognition}},
  year = {2021},
  eprint = {2109.00796},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.00796},
}
Figure