Towards Zero-Shot Sign Language Recognition

Yunus Can Bilge, Ramazan Gokberk Cinbis, Nazli Ikizler-Cinbis

TL;DR

This paper introduces Zero-Shot Sign Language Recognition (ZSSLR), a novel approach to recognize sign language without direct training examples for each sign. By leveraging textual descriptions and attributes from sign language dictionaries, the authors enable models to generalize to unseen signs, paving the way for more scalable sign language recognition systems.

What Problem Does This Paper Address?

Sign Language Recognition (SLR) is a challenging computer vision task due to the subtle visual differences between signs and the vast number of sign languages and their dialects worldwide. Traditional supervised SLR methods require extensive annotated data for every sign, creating a significant data bottleneck. This paper addresses this by proposing Zero-Shot Sign Language Recognition (ZSSLR), which aims to recognize signs from classes that were not seen during training. This is achieved by transferring knowledge from seen classes to unseen classes using auxiliary information like textual descriptions and attributes, effectively bridging the gap between available data and the need for comprehensive sign language understanding.

Methods

The proposed approach employs a two-stream framework for visual representation: one stream processes the full body and hand regions, while the other focuses specifically on hand movements. Spatiotemporal features are extracted using either a Temporal Shift Module (TSM) or a combination of I3D and bi-LSTM networks. For class representation, the authors utilize textual descriptions from sign language dictionaries and attribute-based descriptions. These are encoded using BERT for text and a 53-dimensional attribute vector. A bi-linear compatibility function (Logistic Label Embedding - LLE) is learned to measure the alignment between visual and auxiliary class representations. The paper also introduces techniques to analyze the influence of binary attributes on zero-shot predictions.

Datasets

The authors introduce three new benchmark datasets for ZSSLR:

  • ASL-Text: Derived from the ASLLVD dataset, focusing on the top 250 sign classes and augmented with textual and attribute descriptions.
  • MS-ZSSLR-W(ild): Adapted from the MS-ASL dataset, retaining dialectal variations and including textual and attribute descriptions.
  • MS-ZSSLR-C(lean): Also adapted from MS-ASL, but with manually filtered samples to remove dialects and ensure alignment between visual and textual descriptions.

Key Insights and Ideas

  • Auxiliary Information is Key: Textual descriptions and attributes from dictionaries are powerful sources of knowledge transfer for recognizing unseen sign language classes.
  • Two-Stream Representation Matters: Combining visual information from both the full body and hand regions significantly improves recognition performance, highlighting the importance of fine-grained hand movements.
  • Attributes Enhance Text: While textual descriptions are useful, incorporating attribute-based information leads to substantial improvements, especially when used in conjunction with text.
  • Dialects Pose a Challenge: The presence of sign language dialects, even in training data, can negatively impact zero-shot recognition performance, suggesting a need for more robust handling of variations.
  • Attribute Analysis Provides Insights: Analyzing the influence of specific attributes on predictions can reveal how models learn associations and identify areas for improvement in robustness and explainability.

Key Results

  • The proposed LLE framework with a two-stream representation (body + hand) achieved competitive results on the ASL-Text dataset, reaching a top-1 accuracy of 16.2% and a top-5 accuracy of 43.8%.
  • On the MS-ZSSLR-C dataset, the TSM-based temporal model with combined attribute and text embeddings achieved a top-1 accuracy of 14.7% and a top-5 accuracy of 43.4%.
  • Attribute-based class embeddings generally outperformed text-only embeddings, and combining both sources yielded the best results.
  • The analysis of misclassifications revealed that confused class pairs often share similar hand movements or locations, emphasizing the need for better handshape modeling.

Limitations and Future Work

  • Low Accuracy: The authors acknowledge that the acquired accuracy levels for ZSSLR are relatively low compared to other ZSL domains, indicating a substantial need for further research.
  • Subtle Differences and Dialects: The subtle visual and textual differences between signs, along with the prevalence of dialects, pose significant challenges.
  • Future Work: The authors suggest exploring new ways to obtain and leverage auxiliary knowledge sources, developing more robust visual representations that capture fine details (including facial expressions and pose), and addressing the challenge of sign language dialects. They also highlight the importance of building ZSL models with more robust (implicit) attribute predictors.

Summary updated

Abstract

This paper tackles the problem of zero-shot sign language recognition (ZSSLR), where the goal is to leverage models learned over the seen sign classes to recognize the instances of unseen sign classes. In this context, readily available textual sign descriptions and attributes collected from sign language dictionaries are utilized as semantic class representations for knowledge transfer. For this novel problem setup, we introduce three benchmark datasets with their accompanying textual and attribute descriptions to analyze the problem in detail. Our proposed approach builds spatiotemporal models of body and hand regions. By leveraging the descriptive text and attribute embeddings along with these visual representations within a zero-shot learning framework, we show that textual and attribute based class definitions can provide effective knowledge for the recognition of previously unseen sign classes. We additionally introduce techniques to analyze the influence of binary attributes in correct and incorrect zero-shot predictions. We anticipate that the introduced approaches and the accompanying datasets will provide a basis for further exploration of zero-shot learning in sign language recognition.
BibTeX
@article{bilge2023towards,
  author = {Yunus Can Bilge and Ramazan Gokberk Cinbis and Nazli Ikizler-Cinbis},
  title = {{Towards Zero-Shot Sign Language Recognition}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  url = {https://openreview.net/forum?id=PUVizwFOOzS},
}
Figure