For decades, the standard approach to Sign Language Translation (SLT) has relied on an intermediate bridge: the gloss. A gloss is a written, word-for-sign label (e.g., "HOUSE," "GO," "EAT") used to represent a signed sequence in a text-based format. While glosses allow researchers to treat sign language as a standard machine translation problem, a growing body of research suggests that this bridge is actually a "lossy bottleneck" that strips away the very features that make sign language a complete linguistic system.
The Reductive Nature of Glosses
The fundamental problem with glossing is that it attempts to map a three-dimensional, simultaneous language onto a one-dimensional, sequential string of text. As Amit Moryossef, Kayo Yin, and Yoav Goldberg argue in Including Signed Languages in Natural Language Processing, sign language is not just "hand waving." It relies heavily on non-manual markers—facial expressions, eyebrow movements, mouthings, and body posture—to convey grammatical structure and semantic nuance.
When a researcher labels a sign with a single gloss, these markers are discarded. Mathias Müller and colleagues point out in Considerations for meaningful sign language machine translation based on glosses that glosses are not a natural writing system used by the Deaf community; they are a researcher-invented abstraction. By reducing a complex visual signal to a sequence of tokens, models lose the ability to distinguish between a statement and a question (often marked by eyebrow position) or to understand spatial referencing (where a signer "places" objects in the air to refer to them later).
The Skeletal Information Gap
To avoid the computational weight of raw video, many researchers have turned to skeletal pose estimation—representing the signer as a set of 2D or 3D coordinates. However, this abstraction introduces its own bottleneck. Tejaswini Ananthanarayana demonstrates in A Computational Approach to Sign Language Understanding that human performance drops significantly when interpreting skeleton-only animations compared to raw video.
Current pose-estimation tools often fail to capture:
- Fine-grained finger movements: Essential for distinguishing similar handshapes.
- Facial micro-expressions: Critical for emotional nuance and grammar Patricia Maes.
- Contact points: Amit Moryossef notes that when hands overlap or touch the face, skeletal estimators often "glitch," losing the spatial relationships necessary to identify the sign.
The Shift to Gloss-Free Translation
Recognizing that glosses act as an information bottleneck, the field is moving toward "gloss-free" or end-to-end architectures. In Sign Language Transformers, Necati Cihan Camgöz and Richard Bowden showed that Transformers could effectively handle the non-monotonic mapping between visual features and spoken language without relying solely on intermediate labels.
Furthering this direction, Benjia Zhou and others developed GFSLT-VLP, which uses visual-language pretraining (similar to CLIP) to map video features directly to text. This approach allows the model to "see" the facial expressions and spatial dynamics that glosses ignore. Similarly, the UniGloR framework developed by Eui Jun Hwang and Jong Cheol Park replaces manual glosses with self-supervised spatio-temporal representations, capturing the "essence" of a sign's movement rather than just a text label.
Re-centering Linguistic Reality
The push away from glosses is not just a technical optimization; it is a linguistic necessity. Aashaka Desai and Julie Hochgesang argue in Systemic Biases in Sign Language AI Research that the field has suffered from "technosolutionism"—building tools based on flawed, reductive datasets. They advocate for Deaf-led research that treats sign languages as the complex, multi-dimensional systems they are.
By moving toward models that ingest raw video or high-fidelity 3D motion—as seen in Large Sign Language Models by Sen Zhang—researchers are finally beginning to preserve the rich grammatical and emotional information that was previously lost in translation.