Abstract
Translating between different signed languages is hindered by a severe lack of parallel corpora. This paper presents a novel pipeline that leverages Large Language Models (LLMs) to automatically align independent sign language datasets, creating a multilingual parallel corpus for American (ASL), Chinese (CSL), and German (DGS) sign languages. By employing a multitasking Transformer model that predicts both body poses and textual glosses, the researchers demonstrate significant improvements in sign-to-sign translation quality.
Introduction
While machine translation for spoken languages thrives on massive parallel datasets, signed languages (SLs) remain resource-poor. With over 300 distinct SLs worldwide, the absence of direct translation tools creates significant communication barriers. Existing datasets are typically "uni-language," pairing video with local spoken language text. Manually creating parallel SL-to-SL datasets is prohibitively expensive. This research explores whether AI can bridge this gap by identifying semantic paraphrases across unrelated datasets.
Methodology: The Alignment Pipeline
The authors propose a three-stage approach to build a translation bridge between disparate SL datasets:
1. Paraphrase Alignment
The core innovation lies in using English as a semantic pivot. By translating the textual annotations (glosses) of datasets like How2Sign (ASL), CSL-Daily (CSL), and PHOENIX-14T (DGS) into English, the researchers used Sentence-BERT to identify sentences with near-identical meanings. This process effectively "pairs" videos from different datasets that represent the same concept.
2. Skeletal Representation
To manage computational complexity and preserve privacy, the system extracts 2D skeletal joint positions using tools like OpenPose, which are then lifted into 3D coordinates. This pose-based approach allows the model to focus on the linguistic nuances of movement rather than raw pixel data.
3. Multitasking Translation Model
The translation architecture is an encoder-decoder Transformer. Beyond the standard mapping of source poses to target poses, the model incorporates an auxiliary gloss recognition task. By forcing the model to simultaneously predict textual glosses, it gains a stronger linguistic "anchor," leading to more semantically accurate pose generation.
Data and Results
The alignment algorithm produced a new multilingual parallel dataset with over 3,000 pairs across ASL, CSL, and DGS.
Key Findings
- Performance Gains: The multitasking approach improved ASL → DGS translation, raising the BLEU-4 score from 5.73 to 6.81.
- Cross-Domain Success: The model successfully aligned data across diverse domains, such as instructional ASL videos and German weather broadcasts.
- Linguistic Insights: Analysis showed that ASL signs typically have longer durations than DGS signs, highlighting the importance of temporal re-timing in translation.
Discussion and Future Work
While promising, the pipeline faces challenges like cumulative error propagation and the limitations of text-based metrics (BLEU/ROUGE) for evaluating 3D movement. Future research will focus on "in-the-wild" alignment within similar domains and moving toward realistic avatar generation to enhance user-friendliness for the Deaf community.
Summary updated