Direct Video to Text Sign Language Translation

66 concepts49 papers1 proposal

For years, the standard pipeline for automated Sign Language Translation (SLT) relied on a two-stage process: first, map video frames to a sequence of written word-for-sign labels called glosses, and second, translate those glosses into natural spoken language text. While conceptually straightforward, this architecture introduces a severe linguistic and computational bottleneck.

As Mathias Müller 0002 et al. argue, glosses are an artificial and deeply flawed intermediate representation. Sign languages are multi-modal and simultaneous; a signer conveys meaning through hand shapes, trajectories, facial expressions, and body posture all at once. Reducing this rich, multi-dimensional spatial grammar to a single discrete text token discards vital non-manual signals. Furthermore, annotating glosses frame-by-frame requires expert human linguists, creating a data scarcity crisis that prevents translation models from scaling to the world's diverse sign languages (Abhinav Joshi et al.).

To overcome these limitations, recent research has shifted toward gloss-free sign language translation. This paradigm bypasses symbolic intermediate tokens entirely, training neural networks to map continuous visual sequences directly to natural language text.


Direct Visual-to-Text Mapping

The foundational mechanism of gloss-free translation relies on end-to-end sequence-to-sequence architectures. Instead of forcing the network to make hard, discrete decisions about individual signs along the way, the model passes continuous, high-dimensional visual features directly from a video encoder to a text decoder.

Traditional Pipeline:  [Video] ──> [Discrete Glosses] ──> [Spoken Text]  (Information Loss)
Gloss-Free Pipeline:   [Video] ───────(Continuous)───────> [Spoken Text]  (Preserves Nuance)

In their seminal work on Sign Language Transformers, Necati Cihan Camgoz et al. demonstrated that attention-based architectures can effectively handle the complex, non-monotonic mapping between visual signing and spoken language. By allowing the decoder to attend directly to the encoder's continuous visual representations, the system preserves the subtle semantic cues—such as facial expressions and body shifts—that traditional glosses omit.

This direct mapping unlocks a massive data advantage. As Bowen Shi et al. demonstrate, removing the gloss requirement allows models to learn directly from raw, captioned online videos. This eliminates the reliance on expert-annotated datasets and enables open-domain translation by leveraging the vast amount of unlabelled sign language media available on the internet.


Capturing the Temporal Hierarchy

Without word-for-word glosses to anchor the alignment, a primary challenge emerges: how does a model learn to associate a continuous, fluid video stream with a sequence of words? Sign language operates across multiple temporal scales simultaneously—a rapid finger movement occurs within frames, a single sign spans dozens of frames, and a spatial grammatical structure might influence an entire sentence.

To capture this multi-scale structure without manual labels, Dongxu Li et al. introduced TSPNet (Temporal Semantic Pyramid). TSPNet evaluates video segments at multiple time scales, allowing the model to bind local, fine-grained hand shapes and global, long-term body movements into a unified representation.

Building on this structural intuition, Sobhan Asasi et al. proposed a hierarchical pre-training strategy. By aligning visual features at both the localized segment level and the global video level with target sentences, the model mimics the natural composition of sign language. This structural alignment compensates for the absence of explicit word-level supervision, allowing the network to learn latent grammatical boundaries implicitly.


Learning Semantics via Visual-Language Alignment

How do we ensure that a visual encoder extracts meaningful linguistic features rather than arbitrary visual noise when no glosses are present? The solution lies in Visual-Language Pre-training (VLP), a strategy heavily inspired by contrastive frameworks like CLIP.

The goal of VLP is to pull corresponding video representations and text representations closer together in a shared latent space, while pushing non-matching pairs apart. Benjia Zhou et al. developed GFSLT-VLP, a two-stage pre-training framework that establishes this shared space before fine-tuning the model for translation. This approach proved that gloss-free models could achieve performance parity with traditional, gloss-supervised systems.

[Visual Encoder] ──> Visual Embedding ──┐
                                        ├──> [Contrastive Loss] (Aligns Concepts)
[Text Encoder]   ──> Text Embedding   ──┘

To refine this alignment down to the frame level, Peiqi Jiao et al. introduced Visual Alignment Pre-training (VAP), which greedily aligns video frames directly with text tokens. To ensure the encoder captures both static shapes and dynamic movements, Richard Bowden et al. decoupled the visual stream using dual encoders—one optimized for fine-grained hand shapes and the other for global motion trajectories—guaranteeing that both facets are properly mapped to the target text space.


From Pixels to Skeletal Poses

While raw RGB video contains all the necessary linguistic information, it also introduces significant noise, such as changing backgrounds, lighting conditions, and signer appearance. To isolate the pure linguistic signal, an alternative branch of gloss-free translation extracts and translates 3D skeletal keypoints.

Abhinav Joshi et al. demonstrated with PoseStitch-SLT that end-to-end translation from pose sequences to text can effectively capture sentence-level meaning. By stitching together word-level pose data into millions of grammatically diverse synthetic sentences, the authors bypassed data scarcity and trained a standard Transformer to map raw coordinate sequences directly to text.

Similarly, Eui Jun Hwang et al. introduced UniGloR, a framework that maps skeletal motion to a universal latent space. By learning implicit spatio-temporal features directly from raw keypoint data, the model preserves the continuous, multi-modal nature of sign language without ever needing an intermediate text-based gloss blueprint.


The Inversion: Gloss-Free Sign Language Generation

The intuition behind gloss-free translation—that continuous visual/motion spaces can be mapped directly to natural language—also works in reverse. Traditional Sign Language Production (SLP) systems generated animations by stringing together isolated gloss avatars, resulting in robotic, disjointed movements that lacked natural transitions and facial expressions.

By applying gloss-free principles to generation, models can synthesize continuous sign language directly from text or audio. Fan Qi et al. developed SignGen, an end-to-end latent diffusion model that generates realistic sign language videos directly from text sentences. Because it bypasses the discrete gloss bottleneck, the diffusion process naturally handles co-articulation (the fluid blending of one sign into the next) and preserves contextual non-manual markers.

For 3D avatar generation, Vasileios Baltatzis et al. introduced Neural Sign Actors, which uses an anatomically-informed graph neural network driven by a diffusion model to map text directly to 3D motion sequences. By learning the direct mapping from text semantics to spatial trajectories, the system captures the fluid, simultaneous grammar of natural signing, completing the bidirectional loop of gloss-free sign language processing.

Key Foundational Work

Better Sign Language Translation with STMC-Transformer (2020) — contributed 3 concept elements to this direction.
Multitask training with unlabeled data for end-to-end sign language fingerspelling recognition (2017) — contributed 1 concept elements to this direction.
Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_gloss_free_sign_language_translation_8eb26c83,
  title = {gloss-free sign language translation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gloss-free-sign-language-translation/rdr_8eb26c839a9141b9b2e1f5ff2b6d0d22},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_gloss_free_sign_language_translation_8eb26c83_v1,
  title = {gloss-free sign language translation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gloss-free-sign-language-translation/rdr_8eb26c839a9141b9b2e1f5ff2b6d0d22/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake