Introduction
Sign2GPT is a novel framework designed to bridge the gap between sign language and spoken text without the need for expensive, expert-annotated "gloss" labels. Traditional sign language translation (SLT) has long relied on glosses—written descriptions of signs in their performance order—which act as a bottleneck because they require expert annotators and are unavailable for most sign languages. Sign2GPT achieves state-of-the-art results in "gloss-free" translation by repurposing large pretrained vision and language models through a clever "pseudo-gloss" pretraining strategy.
Methodology and Architecture
The Sign2GPT architecture integrates a frozen vision backbone, a trainable sign encoder, and a frozen Large Language Model (LLM) decoder, connected via lightweight adapters. This modular approach allows the system to leverage the massive amount of knowledge already embedded in general-purpose vision and language models.
- Spatial Backbone: The model uses Dino-V2, a self-supervised Vision Transformer (ViT), to extract high-level spatial features from video frames.
- Sign Encoder: A spatio-temporal transformer processes these features using local self-attention (focusing on a 7-frame window) and temporal downsampling to manage long video sequences efficiently. Research indicated that local attention outperformed global attention, suggesting that sign features are highly localized in time.
- Language Decoder: The framework utilizes XGLM, a 1.7B parameter multilingual GPT model, as the translation engine.
To avoid catastrophic forgetting and minimize computational costs, the authors employ LoRA (Low-Rank Adaptation) and Zero-Gated Cross Attention. This allows the LLM to incorporate sign features gradually without losing its inherent linguistic capabilities. By using "Zero-Gated" attention, the model initially relies on its internal language knowledge and only integrates sign features as training progresses, preserving fluency.
Pseudo-Gloss Pretraining Strategy
The core innovation of Sign2GPT is its pretraining stage, which eliminates the need for manual glossing. The authors use standard NLP tools (like spaCy) to extract "pseudo-glosses" (nouns, verbs, and adjectives) from the target spoken sentences.
These words are converted into "prototypes" using fastText embeddings. The sign encoder is then trained to align video segments with these prototypes. While these pseudo-glosses appear in the spoken word order rather than the sign order, they provide a sufficient signal for the encoder to learn to recognize key concepts within the video. This stage focuses on the existence of concepts rather than their sequence, effectively using NLP as an automated annotator.
Experimental Results
The model was evaluated on two major public benchmarks: RWTH-PHOENIX-Weather 2014T (German Sign Language) and CSL-Daily (Chinese Sign Language). Sign2GPT significantly outperformed previous state-of-the-art (SOTA) gloss-free models:
- Phoenix14T: Achieved a 22.52 BLEU-4 score, a 1.1-point improvement over the previous best gloss-free method.
- CSL-Daily: Achieved a 15.40 BLEU-4 score, representing a substantial 4.4-point jump over previous SOTA.
The efficiency of the adapter-based approach is notable: while the system leverages nearly 1.8 billion parameters, only 16.7 million (less than 1%) are updated during training, making it feasible to train such large-scale systems on standard hardware.
Discussion and Future Work
Sign2GPT demonstrates that linguistic priors in LLMs are highly effective for SLT when properly adapted. However, the authors note that the current pretraining stage ignores the order of signs. While this is partially addressed with positional encodings in the translation stage, a pretraining strategy that accounts for sign-order temporal invariance might be even more powerful.
Future work may explore the application of these techniques to "in-the-wild" video datasets and diverse dialects. Additionally, the localization maps generated during pretraining—which show which part of the video triggered specific word prototypes—could be utilized for automatic sign spotting, providing a new tool for sign language documentation.
Summary updated