Sign languages are fully realized, independent natural languages with their own complex grammar, syntax, and spatial-temporal structures. Translating a sign language video into spoken or written text is not a simple matter of word-for-word substitution. A single sign can change meaning based on facial expressions, body posture, and the speed or trajectory of a movement.
To bridge the massive gap between continuous video frames and natural language text, researchers historically relied on an intermediate representation called a gloss. A gloss is a sequence of written words that maps directly to individual signs in the exact order they are performed. While glosses served as a vital crutch for early computational models, they have become the single largest bottleneck in scaling sign language technologies.
Recent breakthroughs are shifting the field toward Gloss-Free Sign Language Translation (GFSLT)—architectures capable of mapping raw video directly to fluent spoken text without any intermediate human-labeled tokens.
The Gloss Bottleneck
To understand why gloss-free translation is necessary, we must first look at why manual glossing fails to scale.
In a traditional pipeline, expert linguists must watch sign language videos frame-by-frame to mark the precise boundaries of each sign and assign a textual label. As noted by Yutong Chen et al. in A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation, this process is prohibitively expensive and labor-intensive, severely limiting the ability to leverage the vast amounts of unlabelled sign language video available online.
Beyond the sheer cost of labor, glosses are inherently lossy representations. Sign language relies heavily on simultaneous channels of communication, known as multi-articulatory signing. Necati Cihan Camgöz et al. demonstrate in Multi-channel Transformers for Multi-articulatory Sign Language Translation that traditional glosses often fail to capture non-manual features like facial expressions and body poses, stripping away crucial grammatical nuances.
Furthermore, glosses maintain the word order of the sign language, which differs drastically from the grammar of spoken languages. A model trained strictly on glosses must first learn to recognize individual signs in their original order, and then learn a completely separate translation step to reorder those signs into a coherent target sentence.
[Raw Video Stream] ──(Manual Expert Annotation)──> [Gloss Sequence] ──> [Spoken Text]
│
CRITICAL SCALE BOTTLENECK
Gloss-free translation eliminates this middle step entirely, aiming to map the video stream directly to the target text. However, removing the gloss removes the structured supervision that tells a model where a sign begins and ends, and what it means. To make gloss-free systems work, models must learn semantic boundaries and translation rules simultaneously.
Mechanism 1: Visual-Language Pre-training
The most prominent strategy for bypassing manual glosses is Visual-Language Pre-training (VLP). Inspired by contrastive frameworks like CLIP, these methods teach a visual encoder to map video frames into a shared semantic space with a text encoder, using only the final translated sentence as supervision.
To make this alignment precise without frame-level labels, Benjia Zhou et al. introduced GFSLT-VLP in Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining. Their framework uses a two-stage pre-training strategy that aligns visual features directly with spoken text embeddings, proving that gloss-free translation can achieve competitive accuracy on standard benchmarks when backed by robust multimodal pre-training.
[Sign Video Frames] ───> [Visual Encoder] ───┐
├───> [Shared Semantic Space] (Contrastive Loss)
[Target Spoken Text] ──> [Text Encoder] ─────┘
A core challenge in this approach is temporal alignment: a single sentence of spoken text corresponds to a long sequence of video frames. To address this, Peiqi Jiao et al. proposed Visual Alignment Pre-training (VAP) in Visual Alignment Pre-training for Sign Language Translation. VAP greedily aligns video frames with text tokens within a contrastive learning framework, narrowing the performance gap between gloss-free and gloss-based methods.
Similarly, Sobhan Asasi demonstrates in Hierarchical Feature Alignment for Gloss-Free Sign Language Translation that aligning visual features at both the fine-grained segment level and the global video level allows the model to capture both individual gestures and overall sentence meaning without explicit human labeling.
Mechanism 2: Automated Pseudo-Gloss Generation
Rather than completely abandoning the structural benefits of an intermediate representation, another powerful branch of research uses Large Language Models (LLMs) to automate the creation of glosses, removing the human from the loop.
Instead of relying on expert linguists, models can learn from pseudo-glosses generated synthetically from the target text. For example, Ryan Wong et al. developed Sign2GPT in Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation. Their framework leverages frozen, large-scale language models using lightweight adapters trained on automatically generated pseudo-glosses, eliminating the need for frame-by-frame annotations.
Taking this a step further, Jianyuan Guo et al. developed a framework detailed in Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation. Their approach uses an LLM to decompose spoken text into a sequence of base concepts, and then uses visual cues from the video to reorder those concepts into the actual signing sequence. This effectively simulates expert linguistic annotations at a fraction of the cost, providing a structured bridge for end-to-end training.
Mechanism 3: Self-Supervised Spatio-Temporal Representations
A third approach completely replaces text-based intermediate representations with data-driven, self-supervised visual tokens. Instead of forcing a model to map a video frame to a written word, these systems extract the underlying mathematical "essence" of the motion itself.
In A Spatio-Temporal Representation Learning as an Alternative to Traditional Glosses in Sign Language Translation and Production, Eui Jun Hwang et al. introduced UniGloR. Instead of relying on textual glosses, UniGloR learns self-supervised spatio-temporal representations directly from skeletal keypoints. By focusing purely on the geometry and dynamics of the human body, the model learns a robust representation of signs that works symmetrically for both translating sign language to text and producing sign animations from text.
[Raw Video] ──> [Skeletal Keypoints] ──> [Spatio-Temporal Codebook] ──> [Translation Model]
By shifting the target from human-defined words to self-supervised motion tokens, these models sidestep the subjective ambiguity and language-specific constraints of traditional glossing.
Scaling Sign Language AI "Into the Wild"
The ultimate validation of gloss-free translation is its ability to scale to real-world data. Traditional gloss-dependent models are confined to small, highly curated laboratory datasets. Gloss-free architectures, by contrast, can ingest raw internet video directly.
This scalability is demonstrated by Biao Zhang in the creation of YouTube-SL-25, detailed in YouTube-SL-25: A Large-Scale, Open-Domain Multilingual Sign Language Parallel Corpus. By bypassing the requirement for manual verification by native signers, the author scaled data collection to over 3,000 hours of video across 25 different sign languages.
As Linchao Zhu summarizes in Scalable Vision-Language Understanding and Generation, moving away from labor-intensive manual annotations toward end-to-end, hierarchical video modeling is what allows vision-language models to become truly scalable, efficient, and adaptable to real-world applications. By removing the gloss bottleneck, the field is finally unlocking the ability to train robust, open-domain translation systems for signed languages worldwide.
Key Foundational Work
Read My Lips: Continuous Signer Independent Weakly Supervised Viseme Recognition (2014) — contributed 1 concept elements to this direction.
Automatic Signer Diarization - The Mover Is the Signer Approach (2013) — contributed 1 concept elements to this direction.
Sign Language Lexical Recognition With Propositional Dynamic Logic (2013) — contributed 1 concept elements to this direction.