For decades, the field of Sign Language Translation (SLT) relied on a linguistic crutch known as the gloss. A gloss is a written word—usually in all caps—used to label a specific sign (e.g., the sign for "house" is glossed as HOUSE). While glosses provided a convenient intermediate representation between visual video and spoken text, they created a significant information bottleneck.
Researchers are now moving toward gloss-free or end-to-end architectures. This shift treats sign language not as a sequence of discrete word-labels, but as a continuous, multimodal signal where meaning is derived from the simultaneous coordination of hands, face, and body.
The Problem with the Gloss Bottleneck
Traditional SLT pipelines were cascaded: first, a model recognized video frames as glosses, and then a second model translated those glosses into a natural language sentence. As Mathias Müller and colleagues argue in Considerations for meaningful sign language machine translation based on glosses, this approach is linguistically flawed.
Glosses fail to capture "non-manual markers"—the facial expressions, head tilts, and torso movements that function as the grammar and adverbs of sign language. Furthermore, glossing is prohibitively expensive. In low-resource settings like Indian Sign Language, as noted by Abhinav Joshi in the iSign benchmark, the "data drought" of expert-annotated glosses makes traditional training impossible.
Mapping Video Directly to Text
The core intuition of the gloss-free movement is that a model should learn the latent relationship between visual motion and linguistic meaning directly. Kayo Yin demonstrated in Sign Language Translation with Transformers that translating directly from video features can actually outperform translating from "perfect" human-annotated glosses. This suggests that the visual signal contains rich, nuanced information that text-based labels simply discard.
To make this work without the supervision of glosses, researchers use several key mechanisms:
- Joint Recognition and Translation: Necati Cihan Camgoz and Richard Bowden pioneered Sign Language Transformers, which use multi-task learning to pass high-dimensional visual features directly to a decoder, preserving the non-manual information lost in symbolic bottlenecks.
- Visual-Language Pre-training (VLP): Inspired by CLIP, frameworks like GFSLT-VLP by Benjia Zhou and Jun Wan use contrastive learning to align video segments with text tokens. In Visual Alignment Pre-training for Sign Language Translation, Peiqi Jiao shows that greedily aligning frames to text can narrow the performance gap between gloss-free and gloss-based methods.
Leveraging Multimodal LLMs
The recent explosion in Large Language Models (LLMs) has provided a new "bridge" for gloss-free translation. Instead of training a translator from scratch, researchers "prompt" pre-trained models with visual features.
In Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation, Zhigang Chen and colleagues propose FLa-LLM, which uses a frozen LLM to provide linguistic structure while a lightweight visual encoder learns to map signs into the LLM's embedding space. Similarly, Jungeun Kim uses Multimodal LLMs in MMSLT to generate detailed textual descriptions of gestures, which are then distilled into the translation model, effectively replacing manual glosses with automated, descriptive AI "thoughts."
End-to-End Sign Production
The gloss-free philosophy extends to Sign Language Production (SLP)—generating signs from text. Traditional systems translated text to glosses and then glosses to motion, leading to robotic, disjointed movements.
Newer generative models, such as Neural Sign Actors by Vasileios Baltatzis and SignDiff by Sen Fang, use diffusion models to map natural language directly to 3D skeletal sequences or video. By bypassing the gloss bottleneck, these models capture more fluid transitions and preserve the spatial logic of sign language. Fan Qi and Yu Duan demonstrate in SignGen that latent diffusion can generate high-fidelity video that maintains semantic consistency without any intermediate symbolic notation.
The Challenge of Grounding
While gloss-free models are more scalable and expressive, they face a new risk: hallucination. Because these models rely heavily on powerful language backbones (like GPT or LLaMA), they sometimes prioritize generating a fluent sentence over accurately reflecting the visual evidence.
Koel Dutta Chowdhury highlights this in Grounding or Guessing?, finding that gloss-free models are particularly prone to "guessing" based on linguistic probability. Ensuring that these end-to-end systems remain "grounded" in the visual signal remains the primary frontier for the field.