Abstract
Sign Language Translation (SLT) is a challenging cross-modal task that converts visual sign language videos into spoken language text. Traditional high-performing SLT systems rely heavily on "glosses"—intermediate linguistic tokens that represent signs in their natural order. However, gloss annotations require expert linguists, making them expensive and difficult to scale. This paper presents PGG-SLT, a novel framework that utilizes Large Language Models (LLMs) to generate "pseudo glosses." By combining LLM-based text transformation with a video-guided reordering algorithm, PGG-SLT bridges the modality gap without human-annotated glosses. Our approach achieves state-of-the-art results on the Phoenix14T and How2Sign benchmarks, matching or exceeding the performance of models trained on expert labels.
Introduction
The core difficulty in Sign Language Translation (SLT) lies in the significant "modality gap" between visual gestures and the grammatical structure of spoken language. Unlike spoken languages, sign languages often use a different syntax and omit "function words" (e.g., articles like "the" or auxiliary verbs like "is"). Historically, researchers bridged this gap using glosses: a written representation that mirrors the signer's handshapes and sequence (e.g., "ME EAT APPLE" for "I am eating an apple").
While effective, glossing is not a standard written form and requires specialized linguistic knowledge to produce. This creates a scalability bottleneck, as large-scale video-text datasets often lack these annotations. "Gloss-free" methods attempt to map video directly to text but frequently struggle with the complexity of the translation. PGG-SLT addresses this by using LLMs as surrogate experts to generate high-quality intermediate representations automatically.
The PGG-SLT Framework
The PGG-SLT pipeline consists of three distinct stages designed to transition from raw video to fluent text through an intermediate pseudo-gloss representation.
1. LLM-Based Drafting
The first step involves prompting an LLM (such as Gemini 1.5 Pro or Gemma 2) to act as a linguistic translator. Using In-Context Learning (ICL) with a small set of examples (approximately 30 pairs), the LLM learns to transform spoken text into a draft pseudo-gloss. This process involves lemmatizing verbs and removing non-essential grammatical fillers, effectively distilling the semantic core of the sentence into a format that resembles sign language.
2. Video-Guided Reordering
A significant challenge is that LLMs typically output glosses in the order of the source spoken sentence, which may not match the signer's temporal sequence. To correct this, the authors implement a weakly supervised reordering algorithm. A classifier is first trained to identify which pseudo-glosses are present in a given video segment. Then, a "Greedy Reordering Algorithm" aligns these detected tokens with the actual visual flow of the video, ensuring the pseudo-glosses are chronologically accurate.
3. Curriculum Training
The final stage involves a structured training curriculum to stabilize the learning process:
- Stage 1 (Sign2Gloss): A visual encoder is trained to map video frames to the reordered pseudo-glosses using Connectionist Temporal Classification (CTC) loss.
- Stage 2 (Gloss2Sign): A pre-trained translator (e.g., mBART) is fine-tuned to convert these pseudo-glosses back into natural spoken text.
- Stage 3 (Sign2Text): The visual encoder and translator are joined for end-to-end fine-tuning, refining the entire pipeline for maximum translation accuracy.
Experimental Evaluation
The researchers evaluated PGG-SLT on two major benchmarks:
- Phoenix14T: A German Sign Language (DGS) dataset focused on weather reports. This dataset includes ground-truth human glosses, which served as a baseline to validate the quality of the generated pseudo-glosses.
- How2Sign: A large-scale American Sign Language (ASL) dataset. Crucially, How2Sign lacks gloss annotations, making it an ideal testbed for demonstrating the real-world utility of a gloss-free framework.
Key Results and Insights
PGG-SLT demonstrates that LLMs possess sufficient linguistic "common sense" to identify the keywords likely to be signed. The results indicate that temporal alignment is just as important as vocabulary; the reordering step was vital for the model to learn accurate visual-to-semantic mappings.
Performance Benchmarks
On the Phoenix14T dataset, PGG-SLT established a new state-of-the-art for gloss-free models. It achieved a BLEU-4 score of 27.53, significantly outperforming previous leaders such as VAP (26.02) and GFSLT-VLP (21.44). Most impressively, PGG-SLT's performance is competitive with "gloss-based" methods that rely on thousands of expensive human annotations.
On the How2Sign dataset, PGG-SLT also set a new high bar, proving that the framework scales effectively to "in-the-wild" datasets where expert annotations are unavailable.
Conclusion and Future Work
PGG-SLT successfully demonstrates that LLMs can serve as effective surrogate experts for sign language tasks. By automating the generation of intermediate representations, the framework removes the primary barrier to scaling SLT systems.
However, limitations remain. Current pseudo-glosses primarily capture lexical information, potentially missing "non-manual" cues like facial expressions or body posture which carry significant grammatical weight in sign language. Future research may explore multimodal LLMs to generate richer intermediate labels that incorporate these visual nuances, further closing the gap between human and machine sign language translation.
Summary updated