TL;DR
This paper introduces the first publicly available, web-based sign language-to-text dictionary that uses lightweight transformer models for sign language recognition (SLR). By leveraging pose estimation and a novel transformer architecture, they achieve competitive accuracy with significantly fewer parameters, making it suitable for deployment on standard hardware and accessible to the deaf community.
What Problem Does This Paper Address?
Automatic Sign Language Recognition (SLR) has seen significant advancements due to deep learning, yet practical, accessible tools for the deaf community remain scarce. Existing solutions often focus on text-to-sign translation or require specialized hardware. There's a clear need for a system that can translate from sign language to text, improving autonomy for deaf individuals. This paper addresses this gap by creating a sign language-to-text dictionary that is usable, accessible, and runs efficiently on standard devices, specifically for French Belgian Sign Language (LSFB).
Methods
The core of this work is a lightweight transformer-based model for isolated sign language recognition. Instead of processing raw RGB video frames, the system utilizes pose features extracted using MediaPipe. This approach significantly reduces the input dimensionality and mitigates issues related to background variations and lighting conditions.
The model architecture is inspired by the Vision Transformer (ViT) but adapted for sequence classification. It involves:
- Linear Embedding + Positional Encoding: Input pose landmarks are linearly projected and augmented with positional information.
- Classification Token: A special token, similar to ViT, is added to the sequence.
- Transformer Encoder: This core component processes the sequence of tokens.
- MLP Classifier: A final multi-layer perceptron uses the classification token to predict the sign label.
A grid search was performed to optimize meta-parameters like the number of attention heads, encoder layers, token size, and maximum sequence length, balancing performance with model complexity.
Datasets
The primary dataset used is the LSFB dataset, specifically the LSFB-ISOL subset. This dataset consists of 4,181 isolated signs performed by 100 signers, with over 50 hours of video. For this work, the dataset was filtered to include signs with at least 20 examples and French translations, resulting in a final dataset of 700 labels and 77,900 instances. The dataset is split into 70% for training and 30% for testing, with signers held out between sets to ensure generalization.
Key Insights and Ideas
- Pose Estimation as a Preprocessing Step: Using pose landmarks instead of raw RGB frames is a crucial insight. It dramatically reduces computational load and makes the model more robust to environmental variations, effectively "delegating" bias handling to the pose estimation library.
- Lightweight Transformer for SLR: Adapting the transformer architecture, typically used for NLP and image classification, for SLR with pose data proves effective. The key is to remove the decoder part and use a classification token for sequence-level prediction.
- User-Centered Design is Paramount: The success of the dictionary hinges on its usability for the deaf community. The paper emphasizes a user-centered HCI methodology, involving personas, low/high-fidelity prototypes, and continuous evaluation with stakeholders and target users.
- Accessibility and Efficiency are Key: The requirement for the model to run on CPUs and respond within 10 seconds is a significant constraint that drives the choice of a lightweight architecture. This focus on efficiency is what enables widespread adoption.
Key Results
- The best-performing model achieved a top-1 accuracy of 54% and a top-10 accuracy of 83% on the test set for recognizing 700 LSFB signs.
- The model is 10 times lighter (fewer parameters) than other state-of-the-art SLR methods, making it suitable for CPU deployment.
- The system responds in less than 10 seconds to a query, ensuring a fluid user experience.
- The web application has been publicly released and is already adopted by the deaf community.
Limitations and Future Work
- Dataset Limitations: The current dataset is specific to French Belgian Sign Language (LSFB). While the methodology is transferable, new datasets would be needed for other sign languages. The dataset also exhibits class imbalance, leading to better performance on more frequent signs.
- Signer Variability: While signers are held out between train/test sets, the model might still struggle with highly idiosyncratic signing styles or variations not present in the training data.
- Future Work:
- Explore metrics-based methods for recognizing more signs without retraining, potentially by predicting distances between signs.
- Investigate new architectures for improved SLR performance and robustness.
- Implement online learning to leverage user input for model retraining.
- Develop a browser plugin for better integration.
- Conduct further UI design iterations based on user feedback.
Summary updated