How to Align Multiple Signed Language Corpora for Better Sign-to-Sign Translations?

Mert Inan, Yang Zhong, Vidya Ganesh, Malihe AlikhaniNAACL 2025 · DOI

Abstract

Translating between different signed languages is hindered by a severe lack of parallel corpora. This paper presents a novel pipeline that leverages Large Language Models (LLMs) to automatically align independent sign language datasets, creating a multilingual parallel corpus for American (ASL), Chinese (CSL), and German (DGS) sign languages. By employing a multitasking Transformer model that predicts both body poses and textual glosses, the researchers demonstrate significant improvements in sign-to-sign translation quality.

Introduction

While machine translation for spoken languages thrives on massive parallel datasets, signed languages (SLs) remain resource-poor. With over 300 distinct SLs worldwide, the absence of direct translation tools creates significant communication barriers. Existing datasets are typically "uni-language," pairing video with local spoken language text. Manually creating parallel SL-to-SL datasets is prohibitively expensive. This research explores whether AI can bridge this gap by identifying semantic paraphrases across unrelated datasets.

Methodology: The Alignment Pipeline

The authors propose a three-stage approach to build a translation bridge between disparate SL datasets:

1. Paraphrase Alignment

The core innovation lies in using English as a semantic pivot. By translating the textual annotations (glosses) of datasets like How2Sign (ASL), CSL-Daily (CSL), and PHOENIX-14T (DGS) into English, the researchers used Sentence-BERT to identify sentences with near-identical meanings. This process effectively "pairs" videos from different datasets that represent the same concept.

Overview of the Paraphrase Alignment process, showing how independent CSL and DGS datasets are linked via semantic concepts.

2. Skeletal Representation

To manage computational complexity and preserve privacy, the system extracts 2D skeletal joint positions using tools like OpenPose, which are then lifted into 3D coordinates. This pose-based approach allows the model to focus on the linguistic nuances of movement rather than raw pixel data.

Visual tracking of hand and body movements used to generate skeletal pose data.

3. Multitasking Translation Model

The translation architecture is an encoder-decoder Transformer. Beyond the standard mapping of source poses to target poses, the model incorporates an auxiliary gloss recognition task. By forcing the model to simultaneously predict textual glosses, it gains a stronger linguistic "anchor," leading to more semantically accurate pose generation.

Data and Results

The alignment algorithm produced a new multilingual parallel dataset with over 3,000 pairs across ASL, CSL, and DGS.

Key Findings

  • Performance Gains: The multitasking approach improved ASL → DGS translation, raising the BLEU-4 score from 5.73 to 6.81.
  • Cross-Domain Success: The model successfully aligned data across diverse domains, such as instructional ASL videos and German weather broadcasts.
  • Linguistic Insights: Analysis showed that ASL signs typically have longer durations than DGS signs, highlighting the importance of temporal re-timing in translation.

Discussion and Future Work

While promising, the pipeline faces challenges like cumulative error propagation and the limitations of text-based metrics (BLEU/ROUGE) for evaluating 3D movement. Future research will focus on "in-the-wild" alignment within similar domains and moving toward realistic avatar generation to enhance user-friendliness for the Deaf community.

Summary updated

BibTeX
@inproceedings{inan2025how,
  author = {Mert Inan and Yang Zhong and Vidya Ganesh and Malihe Alikhani},
  title = {{How to Align Multiple Signed Language Corpora for Better Sign-to-Sign Translations?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.202},
}
Figure