Abstract
Vision-based sign language recognition is an extensively researched problem aimed at advancing communication between deaf and hearing individuals. Numerous Sign Language Recognition (SLR) datasets have been introduced to promote research in this field, spanning multiple languages, vocabulary sizes, and signers. However, most existing popular datasets focus predominantly on the frontal view of signers, neglecting visual information from other perspectives. In practice, many sign languages contain words that have similar hand movements and expressions, making it challenging to differentiate between them from a single frontal view. Although a few studies have proposed sign language datasets using multi-view data, these datasets remain limited in vocabulary size and scale, hindering their generalizability and practicality. To address this issue, we introduce a new large-scale, multi-view sign language recognition dataset spanning 1,000 glosses and 30 signers, resulting in over 84,000 multi-view videos. To the best of our knowledge, this is the first multi-view sign language recognition dataset of this scale. In conjunction with offering a comprehensive dataset, we perform extensive experiments to assess the performance of state-of-the-art Sign Language Recognition models utilizing on our dataset. The findings indicate that utilizing multi-view data substantially enhances model accuracy across all models, with a maximum performance improvement of up to 19.75% compared to models trained on single-view data. Our dataset and baseline models are publicly accessible on GitHub.
Introduction: The Ambiguity of Single-View Recognition
Most current Sign Language Recognition (SLR) systems rely on a single frontal camera view. While convenient, this approach has a major flaw: sign language is inherently three-dimensional. Many signs involve subtle hand movements toward or away from the body that are obscured or "flattened" when viewed only from the front.
Visual ambiguity leads to high error rates in real-world applications. For example, a sign involving a hand-shaking motion might look identical to a static hand gesture from a frontal perspective. While a few multi-view datasets existed previously, they were too small in vocabulary (often under 200 words) to train robust, generalizable deep learning models.
Methodology
The researchers developed a rigorous pipeline to create and evaluate the Multi-VSL dataset, ensuring high data quality and signer privacy.
The pipeline consists of four main stages:
- Data Collection: A synchronized three-camera setup was used to capture signs from Left (~45°), Front (~0°), and Right (~45°) angles.
- Standardization: All videos were converted to 30 FPS MP4 format.
- Annotation & Splitting: A double-check process involving human annotators and YOLOv9-based automated pose detection was used to split glosses and crop human poses, reducing background noise.
- De-identification: To protect signer privacy, faces were blurred using Gaussian filters while maintaining the clarity of hand and body movements.
To benchmark the data, the authors adapted four state-of-the-art models (I3D, Swin Transformer, MViTv2, and VTNPF) using a multi-view framework. This involved initializing three identical encoders (one per view), concatenating their latent features, and using a Multilayer Perceptron (MLP) for final classification. They also introduced a "Partial Training" strategy to ensure model robustness even when one or more camera views are missing during inference.
The Multi-VSL Dataset
The primary contribution of this work is the Multi-VSL (Multi-view Vietnamese Sign Language) dataset. It represents a significant leap in scale for multi-view SLR research.
As shown in the comparison above, Multi-VSL features 1,000 glosses, 30 signers, and 84,764 videos. This scale is substantially larger than previous multi-view datasets like KL_MV2DSL (200 glosses) or MVSL (50 glosses), providing the necessary data volume for modern transformer-based architectures.
Key Results and Insights
The evaluation of Multi-VSL yielded several critical insights:
- Significant Accuracy Boost: Utilizing three views instead of one improved the Top-1 accuracy of the I3D model by 19.75% on the 1,000-gloss dataset.
- Effective Disambiguation: The multi-view approach successfully solved the "ambiguity problem." For gloss pairs like "papers" and "blanket"—which are frequently confused in frontal views—the multi-view model reduced the confusion ratio to 0%.
- Superiority of Pose + RGB: The VTNPF model, which utilizes both raw video frames and skeletal pose data, consistently outperformed RGB-only models. It achieved 76.32% Top-1 accuracy on the full 1,000-gloss dataset.
- Robustness through Augmentation: Models trained with "Partial training" (simulating missing views) significantly outperformed single-view baselines even when 50% of the camera data was missing at test time.
Conclusion and Future Work
Multi-VSL addresses a critical gap in SLR research by providing a large-scale, multi-view dataset that effectively mitigates visual ambiguity. While currently focused on Vietnamese Sign Language and fixed camera angles, future work will explore more sophisticated fusion techniques and "view-invariant" representations to generalize recognition to arbitrary camera positions in "in-the-wild" scenarios.
Summary updated