Sign Language Recognition: A Large-Scale Multi-View Dataset and Comprehensive Evaluation

Son Dinh Nguyen, Tuan Dung Nguyen, Tri Tran, Nguyen Dang Huy Pham, Thuan Hieu Tran, Ngoc Anh Tong, Hoang Quang Huy, Phi Le NguyenWACV 2025 · OpenReview

Abstract

Vision-based sign language recognition is an extensively researched problem aimed at advancing communication between deaf and hearing individuals. Numerous Sign Language Recognition (SLR) datasets have been introduced to promote research in this field, spanning multiple languages, vocabulary sizes, and signers. However, most existing popular datasets focus predominantly on the frontal view of signers, neglecting visual information from other perspectives. In practice, many sign languages contain words that have similar hand movements and expressions, making it challenging to differentiate between them from a single frontal view. Although a few studies have proposed sign language datasets using multi-view data, these datasets remain limited in vocabulary size and scale, hindering their generalizability and practicality. To address this issue, we introduce a new large-scale, multi-view sign language recognition dataset spanning 1,000 glosses and 30 signers, resulting in over 84,000 multi-view videos. To the best of our knowledge, this is the first multi-view sign language recognition dataset of this scale. In conjunction with offering a comprehensive dataset, we perform extensive experiments to assess the performance of state-of-the-art Sign Language Recognition models utilizing on our dataset. The findings indicate that utilizing multi-view data substantially enhances model accuracy across all models, with a maximum performance improvement of up to 19.75% compared to models trained on single-view data. Our dataset and baseline models are publicly accessible on GitHub.

Introduction: The Ambiguity of Single-View Recognition

Most current Sign Language Recognition (SLR) systems rely on a single frontal camera view. While convenient, this approach has a major flaw: sign language is inherently three-dimensional. Many signs involve subtle hand movements toward or away from the body that are obscured or "flattened" when viewed only from the front.

Visual ambiguity leads to high error rates in real-world applications. For example, a sign involving a hand-shaking motion might look identical to a static hand gesture from a frontal perspective. While a few multi-view datasets existed previously, they were too small in vocabulary (often under 200 words) to train robust, generalizable deep learning models.

Methodology

The researchers developed a rigorous pipeline to create and evaluate the Multi-VSL dataset, ensuring high data quality and signer privacy.

The Multi-VSL data processing pipeline, from collection to de-identification.

The pipeline consists of four main stages:

  1. Data Collection: A synchronized three-camera setup was used to capture signs from Left (~45°), Front (~0°), and Right (~45°) angles.
  2. Standardization: All videos were converted to 30 FPS MP4 format.
  3. Annotation & Splitting: A double-check process involving human annotators and YOLOv9-based automated pose detection was used to split glosses and crop human poses, reducing background noise.
  4. De-identification: To protect signer privacy, faces were blurred using Gaussian filters while maintaining the clarity of hand and body movements.

To benchmark the data, the authors adapted four state-of-the-art models (I3D, Swin Transformer, MViTv2, and VTNPF) using a multi-view framework. This involved initializing three identical encoders (one per view), concatenating their latent features, and using a Multilayer Perceptron (MLP) for final classification. They also introduced a "Partial Training" strategy to ensure model robustness even when one or more camera views are missing during inference.

The Multi-VSL Dataset

The primary contribution of this work is the Multi-VSL (Multi-view Vietnamese Sign Language) dataset. It represents a significant leap in scale for multi-view SLR research.

Comparison of Multi-VSL with existing SLR datasets across various metrics.

As shown in the comparison above, Multi-VSL features 1,000 glosses, 30 signers, and 84,764 videos. This scale is substantially larger than previous multi-view datasets like KL_MV2DSL (200 glosses) or MVSL (50 glosses), providing the necessary data volume for modern transformer-based architectures.

Key Results and Insights

The evaluation of Multi-VSL yielded several critical insights:

  • Significant Accuracy Boost: Utilizing three views instead of one improved the Top-1 accuracy of the I3D model by 19.75% on the 1,000-gloss dataset.
  • Effective Disambiguation: The multi-view approach successfully solved the "ambiguity problem." For gloss pairs like "papers" and "blanket"—which are frequently confused in frontal views—the multi-view model reduced the confusion ratio to 0%.
  • Superiority of Pose + RGB: The VTNPF model, which utilizes both raw video frames and skeletal pose data, consistently outperformed RGB-only models. It achieved 76.32% Top-1 accuracy on the full 1,000-gloss dataset.
  • Robustness through Augmentation: Models trained with "Partial training" (simulating missing views) significantly outperformed single-view baselines even when 50% of the camera data was missing at test time.

Conclusion and Future Work

Multi-VSL addresses a critical gap in SLR research by providing a large-scale, multi-view dataset that effectively mitigates visual ambiguity. While currently focused on Vietnamese Sign Language and fixed camera angles, future work will explore more sophisticated fusion techniques and "view-invariant" representations to generalize recognition to arbitrary camera positions in "in-the-wild" scenarios.

Summary updated

Abstract

Vision-based sign language recognition is an extensively researched problem aimed at advancing communication be- tween deaf and hearing individuals. Numerous Sign Lan- guage Recognition (SLR) datasets have been introduced to promote research in this field, spanning multiple languages, vocabulary sizes, and signers. However, most existing pop- ular datasets focus predominantly on the frontal view of signers, neglecting visual information from other perspec- tives. In practice, many sign languages contain words that have similar hand movements and expressions, making it challenging to differentiate between them from a single frontal view. Although a few studies have proposed sign lan- guage datasets using multi-view data, these datasets remain limited in vocabulary size and scale, hindering their gener- alizability and practicality. To address this issue, we in- troduce a new large-scale, multi-view sign language recog- nition dataset spanning 1,000 glosses and 30 signers, re- sulting in over 84,000 multi-view videos. To the best of our knowledge, this is the first multi-view sign language recognition dataset of this scale. In conjunction with of- fering a comprehensive dataset, we perform extensive ex- periments to assess the performance of state-of-the-art Sign Language Recognition models utilizing on our dataset. The findings indicate that utilizing multi-view data substantially enhances model accuracy across all models, with a maxi- mum performance improvement of up to 19.75% compared to models trained on single-view data. Our dataset and baseline models are publicly accessible on GitHub
BibTeX
@inproceedings{nguyen2025sign,
  author = {Son Dinh Nguyen and Tuan Dung Nguyen and Tri Tran and Nguyen Dang Huy Pham and Thuan Hieu Tran and Ngoc Anh Tong and Hoang Quang Huy and Phi Le Nguyen},
  title = {{Sign Language Recognition: A Large-Scale Multi-View Dataset and Comprehensive Evaluation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=QDOts81YSA},
}
Figure