TL;DR
This paper introduces "Logos," a massive new dataset for Russian Sign Language (RSL) that includes a novel annotation of "visually similar signs" (VSSigns). They demonstrate that pre-training models on Logos significantly improves performance on other sign language recognition tasks, especially in low-resource scenarios, and that explicitly grouping VSSigns in the dataset leads to better model encoders.
What Problem Does This Paper Address?
Sign Language Recognition (SLR) is crucial for communication accessibility but faces two major hurdles: a scarcity of diverse, large-scale datasets for individual sign languages, and the inherent ambiguity of signs that look similar but have different meanings (visually similar signs or VSSigns). Existing datasets are often limited in size, vocabulary, or the number of signers, making it difficult to train robust models or transfer knowledge across languages. Furthermore, how to best handle VSSigns in datasets – whether to group them or treat them as distinct – is an open question with implications for model performance.
Methods
The core of this work involves introducing the Logos dataset and leveraging it for cross-lingual transfer learning.
- Logos Dataset: This is a new, large-scale RSL dataset collected via crowdsourcing. A key innovation is the explicit annotation of visually similar signs (VSSigns), grouping signs that have the same manual components but different meanings. This grouping is done through a multi-stage process involving model predictions and expert validation.
- Transfer Learning: The researchers investigate using Logos as a pre-training resource. They compare two main strategies:
- Pre-train and Fine-tune: Training a model on Logos and then fine-tuning it on a target low-resource dataset.
- Multi-dataset Co-training: A more advanced approach where models are trained jointly on multiple datasets (Logos plus target datasets) using language-specific classification heads. This allows the model to learn from diverse sign languages simultaneously.
- VSSigns Grouping Impact: They specifically analyze how the VSSigns grouping in Logos affects model performance, both for general recognition and for transfer learning.
Datasets
- Logos (New Dataset): A large-scale Russian Sign Language (RSL) dataset.
- Size: 199,668 videos, 221.4 hours.
- Signers: 381 signers.
- Vocabulary: 2,863 glosses, grouped into 2,004 VSSign labels.
- Key Feature: Explicit annotation of visually similar sign groups (VSSigns).
- Target Datasets for Transfer Learning:
- AUTSL: Turkish Sign Language dataset.
- WLASL: American Sign Language dataset.
- Additional Pre-training Baseline:
- MM-WLAuslan: Australian Sign Language dataset.
Key Insights and Ideas
- Logos is a Game-Changer for RSL: The sheer scale and diversity of Logos make it a valuable resource for RSL research, and its VSSign annotation offers a novel way to handle sign ambiguity.
- Pre-training on Logos is Universally Beneficial: A model pre-trained on Logos acts as a strong "universal encoder," significantly boosting performance on other sign languages, even with very limited target data (few-shot learning). This suggests Logos captures fundamental visual features of sign language.
- VSSigns Grouping Matters: Explicitly grouping visually similar signs in the dataset (as done in Logos) demonstrably improves the quality of the learned visual encoder, leading to better downstream performance. This is a crucial finding for dataset design.
- Multi-dataset Co-training Outperforms Simple Transfer: Jointly training on Logos and target datasets with language-specific heads is more effective than standard pre-train and fine-tune, especially for low-resource languages.
Key Results
- State-of-the-Art Performance: The Logos pre-trained model achieved superior results on the WLASL dataset and competitive results on the AUTSL dataset, using only RGB video input.
- Few-Shot Learning Success: Even with as few as 1-3 samples per class, transfer learning from Logos provided substantial accuracy gains on AUTSL and WLASL, demonstrating its effectiveness in data-scarce scenarios.
- VSSigns Grouping Benefits: Training with grouped VSSigns annotations on Logos led to higher accuracy compared to using ungrouped labels, both in baseline training and in transfer learning scenarios.
- Encoder Generalization: The study confirmed that the encoder pre-trained on Logos learns general sign embeddings, not just language-specific features, as evidenced by the superior performance of direct classification over label mapping.
Limitations and Future Work
- Architecture Specificity: The current work is limited to the MViT baseline architecture. Future work could explore other model architectures.
- Downstream Task Scope: The focus was on Isolated Sign Language Recognition (ISLR). Extending this to continuous sign language recognition and translation would be valuable.
- Dataset Diversity: Logos is exclusively for RSL. While it shows broad transferability, future research could explore datasets from even more diverse linguistic backgrounds.
- Dataset Demographics: The Logos dataset reflects the demographics of its crowdsourced participants, leading to an unbalanced distribution in age and gender. While this is acknowledged, it limits direct applicability to more diverse settings.
- Potential Misuse: The authors acknowledge the potential for misuse of the dataset, such as identifying individuals or enabling surveillance, despite making it available for research purposes.
Summary updated