Continuous Bangla Sign Language Translation: Mitigating the Expense of Gloss Annotation with the Assistance of Graph

Safaeid Hossain Arib, Rabeya Akter, Sejuti RahmanarXiv 2025 · OpenReview

TL;DR

This paper tackles the challenge of translating Bangla Sign Language (BdSL) into spoken language, aiming to bridge communication gaps for the deaf and hard-of-hearing community. They propose a novel "gloss-free" approach that fuses Transformer and Spatio-Temporal Graph Convolutional Network (STGCN)-LSTM architectures, achieving state-of-the-art results on multiple datasets and demonstrating improved performance without relying on costly gloss annotations.

What Problem Does This Paper Address?

Millions worldwide are deaf or hard of hearing, and sign language is their primary means of communication. However, in societies that prioritize spoken languages, sign language often faces underestimation, leading to communication barriers and social exclusion. While sign language translation (SLT) technology aims to alleviate this, current methods often struggle with the complexity of sign language, which involves not just hand gestures but also facial expressions and body movements. A significant hurdle in developing accurate SLT systems is the need for "gloss annotations" – textual representations of sign gestures – which are expensive and time-consuming to create. This paper addresses the need for more accessible and effective SLT, particularly for Bangla Sign Language (BdSL), by developing a "gloss-free" translation system that mitigates the cost and complexity associated with gloss annotation.

Methods

The core of this research lies in a novel architectural fusion that combines the strengths of two powerful deep learning models:

  • Transformer Architecture: Known for its success in sequence-to-sequence tasks, the Transformer excels at capturing contextual information and long-range dependencies within the sign language video.
  • Spatio-Temporal Graph Convolutional Network (STGCN)-LSTM: This component is designed to capture the inherent spatial and temporal relationships within sign language. It leverages keypoint data (extracted from human body joints) to build a spatio-temporal graph structure, allowing the model to understand the dynamic movements and poses involved in signing.

The proposed method fuses these two streams:

  1. Video Encoding: An I3D network extracts features from video frames, which are then positionally encoded and processed by a Transformer encoder.
  2. Keypoint Encoding: MediaPipe is used to extract 33 pose landmarks from the human body. These keypoints are then fed into an STGCN-LSTM encoder to capture spatio-temporal dynamics.
  3. Two-Stream Fusion: The outputs from the video and keypoint encoding streams are fused (specifically, using summation) to create a comprehensive representation.
  4. Transformer Decoder: This fused representation is then passed to a Transformer decoder to generate the spoken language translation.

This "gloss-free" approach bypasses the need for manual gloss annotations, making the translation process more efficient and cost-effective.

Datasets

The authors evaluated their model on four diverse sign language datasets:

  • RWTH-PHOENIX-2014T: A continuous German Sign Language (DGS) dataset.
  • CSL-Daily: A dataset of Chinese Sign Language (CSL) covering daily life events.
  • How2Sign: A large-scale, multimodal dataset for American Sign Language (ASL) translation.
  • BornilDB v1.0: A newly introduced dataset for Bangla Sign Language (BdSL). This dataset is notable for its dynamic backgrounds and varied video configurations, presenting a unique challenge compared to the other datasets.

The paper also introduces benchmarking on the BornilDB v1.0 dataset for the first time.

Key Insights and Ideas

  • Fusion is Key: The paper strongly advocates for the fusion of different modalities (video features and skeletal keypoints) and architectures (Transformer and STGCN-LSTM) to achieve robust sign language translation. This combination captures both contextual understanding and the crucial spatio-temporal dynamics of signing.
  • Gloss-Free is the Future: By demonstrating state-of-the-art performance without relying on expensive gloss annotations, the research highlights the viability and importance of gloss-free SLT for broader accessibility and scalability.
  • STGCN for Spatio-Temporal Dynamics: The STGCN-LSTM component effectively captures the intricate movements and poses of sign language, which are critical for accurate translation and are often missed by purely sequence-based models.
  • BornilDB as a Challenging Benchmark: The introduction of BornilDB v1.0 as a benchmark dataset, with its dynamic backgrounds and varied video conditions, pushes the boundaries of SLT research and encourages the development of more robust models.

Key Results

The proposed method achieved state-of-the-art performance across multiple datasets:

  • RWTH-PHOENIX-2014T: Achieved a BLEU-4 score of 19.75, surpassing existing methods.
  • CSL-Daily: Achieved a BLEU-4 score of 6.14, showing significant improvement.
  • How2Sign: Outperformed the baseline with BLEU-4 scores of 9.9 (Validation) and 8.53 (Test).
  • BornilDB v1.0: Demonstrated superior performance with BLEU-4 scores of 0.72 (Validation) and 0.58 (Test), establishing a new benchmark for this dataset.

Qualitative results showed impressive accuracy in capturing intended meaning, with predicted sentences closely mirroring references and effectively conveying meaning even with slight wording variations.

Limitations and Future Work

  • Computational Resources: The authors acknowledge that current models, including theirs, demand substantial computational resources.
  • Future Work:
    • Develop a more lightweight version of the system for mobile devices with lower resource requirements.
  • Strengthen the system's resilience and facilitate real-time processing of video data beyond pre-recorded inputs.
  • Augment the practical utility of the system in real-world scenarios, making it more responsive and adaptable to diverse communication situations.
  • Continue to improve SLT performance, as current scores still fall short of declaring the task as fully solved.

Summary updated

Abstract

Millions of individuals worldwide are affected by deafness and hearing impairment. Sign language serves as a sophisticated means of communication for the deaf and hard of hearing. However, in societies that prioritize spoken languages, sign language often faces underestimation, leading to communication barriers and social exclusion. The Continuous Bangla Sign Language Translation project aims to address this gap by enhancing translation methods. While recent approaches leverage transformer architecture for state-of-the-art results, our method integrates graph-based methods with the transformer architecture. This fusion, combining transformer and STGCN-LSTM architectures, proves more effective in gloss-free translation. Our contributions include architectural fusion, exploring various fusion strategies, and achieving a new state-of-the-art performance on diverse sign language datasets, namely RWTH-PHOENIX-2014T, CSL-Daily, How2Sign, and BornilDB v1.0. Our approach demonstrates superior performance compared to current translation outcomes across all datasets, showcasing notable improvements of BLEU-4 scores of 4.01, 2.07, and 0.5, surpassing those of GASLT, GASLT and slt_how2sign in RWTH-PHOENIX-2014T, CSL-Daily, and How2Sign, respectively. Also, we introduce benchmarking on the BornilDB v1.0 dataset for the first time. Our method sets a benchmark for future research, emphasizing the importance of gloss-free translation to improve communication accessibility for the deaf and hard of hearing.
BibTeX
@misc{arib2025continuous,
  author = {Safaeid Hossain Arib and Rabeya Akter and Sejuti Rahman},
  title = {{Continuous Bangla Sign Language Translation: Mitigating the Expense of Gloss Annotation with the Assistance of Graph}},
  year = {2025},
  url = {https://openreview.net/forum?id=dwjIWuPR6a},
}
Figure