Bridging the Modality Gap in Sign Language Translation

This direction is split into other directions: cross-modal grammatical divergence. Its last version stays readable and citable here.

Version 1 of 174 papers82 conceptsfrozen 2026-10-04

To the uninitiated, sign language might look like a manual version of a spoken language—English words replaced by hand gestures. If this were true, Sign Language Translation (SLT) would be a simple mapping problem. However, as Kazuko Takahashi notes in An Approach to Machine Translation from Japanese Text to JSL Text, true sign languages like Japanese Sign Language (JSL) or American Sign Language (ASL) are independent linguistic systems with their own complex grammars, 3D spatial logic, and non-linear syntax.

The core challenge in this field is the "modality gap": the fundamental difficulty of mapping continuous, multi-channel 3D visual signals to discrete, linear text.

The Myth of Word-for-Sign Correspondence

The most persistent hurdle in SLT is that sign languages do not follow the word order or grammatical rules of spoken languages. In Sign Language Production: A Review, Razieh Rastgoo and Sergio Escalera explain that providing text captions is often insufficient for the Deaf community because sign languages involve complex conceptual re-ordering.

Consider "connectives" like and, but, or because. In spoken Italian, these are explicit words. But in Italian Sign Language (LIS), Camillo Lugaresi and Barbara Di Eugenio found in Translating Italian connectives into Italian Sign Language that these functions are often "absorbed" into spatial positioning or facial expressions. This creates a non-monotonic relationship: the order of visual signs does not correspond linearly to the order of spoken words.

The Gloss as a Linguistic Anchor

Because of this structural mismatch, researchers often use an intermediate representation called a gloss—a written label (usually in all caps) that represents a specific sign's meaning without its spoken-language grammar.

For years, the standard pipeline has been:

  1. Recognition: Video $\rightarrow$ Glosses
  2. Translation: Glosses $\rightarrow$ Spoken Text

In Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation, researchers demonstrated that training models to perform both tasks simultaneously helps the system learn the complex, non-linear mapping between visual features and semantic meaning. However, glosses are a double-edged sword. As Malihe Alikhani argues in Including Signed Languages in Natural Language Processing, glosses are an imperfect, coarse approximation of the rich, multi-channel information (hands, face, body) present in actual signing.

Overcoming the Data Drought

The biggest bottleneck in SLT is the lack of large-scale, annotated datasets. While text-to-text translation benefits from trillions of tokens, sign language data is "extremely low-resource."

To solve this, researchers are turning to synthetic data and "gloss-free" methods:

Why It Works: Moving Beyond Vision

The field is shifting from treating sign language as a computer vision problem (recognizing gestures) to a natural language processing problem (understanding spatial grammar).

Newer architectures like SONAR-SLT by Yasser Hamidullah and Cristina España-Bonet use language-agnostic sentence embeddings. By mapping sign language videos into the same mathematical space as hundreds of spoken languages, these models can translate signs into languages they weren't even trained on.

As Richard Bowden and colleagues highlight in the WMT-SLT23 Shared Task, while we have scaled data by 25x, the "modality gap" remains wide. The future of the field lies in models that don't just see the hands, but understand the 3D linguistic logic that governs them.

Papers in this version

Showing 60 of 74 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bridging_the_modality_gap_in_sign_langua_59939d05,
  title = {Bridging the Modality Gap in Sign Language Translation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-the-modality-gap-in-sign-language-translation/rdr_59939d0566c14dccb6beaaa78bfb41b6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_bridging_the_modality_gap_in_sign_langua_59939d05_v1,
  title = {Bridging the Modality Gap in Sign Language Translation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-the-modality-gap-in-sign-language-translation/rdr_59939d0566c14dccb6beaaa78bfb41b6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}