When we translate between two spoken languages—such as English and Spanish—we manage differences in vocabulary, word order, and idioms. However, both languages share a fundamental medium: they are linear, auditory-vocal systems.
Translating between a spoken language and a sign language introduces a much deeper challenge: cross-modal structural divergence. Sign languages are not visual encodings of spoken words, nor do they follow the grammatical rules of the spoken languages used in the same geographic regions. Instead, they are independent natural languages with distinct visual-gestural grammars. They leverage three-dimensional space, simultaneous expressions (combining hands, face, and body posture), and non-monotonic structures that fundamentally break the assumptions of standard sequence-to-sequence machine translation models.
To build systems that accurately translate between speech/text and sign language, researchers must design architectures that explicitly reconcile these divergent linguistic frameworks.
The Anatomy of the Modality Gap
Why does a direct, word-for-sign linear mapping fail? The divergence manifests in three primary ways: non-monotonicity, structural absorption, and spatial syntax.
Non-Monotonicity and Reordering
In spoken languages, information is delivered sequentially over time. In sign languages, the chronological order of signs frequently departs from the word order of the corresponding spoken sentence. For example, sign languages often employ topicalization, where the object or topic of a sentence is established at the beginning, regardless of its position in the spoken source.
As Necati Cihan Camgoz et al. demonstrate in Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation, this non-monotonic relationship prevents standard sequential models from relying on local alignments. Traditional sequence models assume a relatively linear mapping, but sign translation requires capturing long-range, non-local dependencies to successfully reorder concepts.
Structural Absorption
Spoken languages rely heavily on explicit structural words—such as prepositions, conjunctions, and articles—to establish syntactic relationships. Sign languages often omit these discrete tokens entirely, opting instead for the "economy" of visual communication.
The mechanism behind this is explored by Camillo Lugaresi and Barbara Di Eugenio in Translating Italian connectives into Italian Sign Language. They show that spoken connectives are frequently absorbed into the spatial morphology or non-manual markers (such as facial expressions or head tilts) of surrounding signs. A translation model cannot simply look for a sign corresponding to "because" or "under"; it must learn how an abstract linguistic function is re-encoded across entirely different sensory channels.
Spatial Syntax
Perhaps the most distinct element of sign language grammar is its use of three-dimensional space to track entities and resolve references. In a spoken language, we use pronouns ("he", "she", "it") to maintain discourse continuity. A signer will instead establish a physical location in the signing space for an entity and reference that entity later by directing signs or glances back to that coordinate.
Spoken English: "The boy met the girl and he gave a book to her."
Spatial Syntax: [Assign BOY to Left] [Assign GIRL to Right] [Action GIVE moves Left-to-Right]
Standard natural language processing tools are designed for linear text or speech, making them fundamentally blind to this spatial logic. As Kenneth DeHaan and Malihe Alikhani point out in Signed Coreference Resolution, resolving who is doing what to whom requires models that can track these spatial-temporal heuristics in a 3D coordinate space rather than relying on text-based discourse analysis.
The Evolution of Structural Mapping
To bridge this structural divide, translation architectures have evolved from rigid, rule-based systems to flexible, data-driven neural frameworks.
Explicit Semantic Transformation
Early translation systems relied on human-engineered rules to map spoken syntax to sign syntax. For instance, Ian Marshall and Éva Sáfár developed A Prototype Text to British Sign Language (BSL) Translation System, which focused on transforming underlying semantic representations rather than surface-level words. These systems explicitly altered verb arguments and generated spatial parameters to satisfy the grammatical requirements of the target sign language. Similarly, R. San-Segundo et al. in A Spanish speech to sign language translation system for assisting deaf-mute people used rule-based structures to handle systematic variations, such as introducing mandatory subjects in Spanish Sign Language that are frequently dropped in spoken Spanish.
The Gloss Bottleneck
As the field shifted toward deep learning, researchers introduced an intermediate representation known as glosses—written words capitalized to represent individual signs in the chronological order they are performed.
Spoken Text: "I am going to the market."
Gloss Sequence: MARKET I GO.
Glosses served as a vital bridge, allowing researchers to break the problem into two parts: video-to-gloss (recognition) and gloss-to-text (translation). However, glosses introduce a severe linguistic bottleneck. As Yong Cao and Wei Li note in Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation, glosses compress the vocabulary space drastically and strip away the rich, simultaneous non-manual expressions occurring on the face and body.
Furthermore, treating gloss translation as a standard machine translation task remains difficult due to data scarcity. Kevin Duh and Xuan Zhang 0008 demonstrate in Approaching Sign Language Gloss Translation as a Low-Resource Machine Translation Task that because gloss sequences preserve unique syntactic structures like topicalization, models require domain-aware optimization to translate them into natural, grammatically correct spoken language.
Modern Solutions: Factorization and Gloss-Free Alignment
To overcome the limitations of glosses while still managing cross-modal divergence, modern architectures employ two primary strategies: explicit task factorization and gloss-free representation learning.
Decoupling Selection and Reordering
For the reverse task—Sign Language Production (generating signs from text)—models must learn exactly how to transform spoken syntax into sign syntax. Harry Walsh addresses this in Select and Reorder: A Novel Approach for Neural Sign Language Production by factorizing the translation pipeline into two distinct steps:
- Lexical Selection: Choosing the correct target vocabulary tokens based on the source text.
- Structural Reordering: Permuting those tokens to match the non-monotonic grammar of the sign language.
By isolating the structural transformation from the vocabulary mapping, the model learns the underlying geometric reordering rules more efficiently, even when training data is highly constrained.
Gloss-Free Translation via Large Language Models
The latest wave of research aims to eliminate manual gloss annotations entirely by leveraging the implicit structural understanding of Large Language Models (LLMs).
Because LLMs excel at manipulating syntax, they can be used to simulate the grammatical reordering process. Jianyuan Guo and Peike Li demonstrate this in Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation. Their framework uses an LLM to generate "pseudo-glosses" from spoken text, which are then dynamically reordered using visual cues from the video.
Similarly, Ryan Wong et al. introduce Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation, which connects visual sign encoders directly to frozen language models via lightweight adapters. The underlying intuition is that the language model acts as a powerful grammatical anchor; the visual adapter's role is simply to map the continuous, non-linear physical movements into the semantic and syntactic space that the language model already understands.
Ultimately, whether through explicit rule-based transformations, factorized neural layers, or LLM-driven alignment, solving sign language translation requires moving past simple token substitution. Success relies on directly modeling the structural divergence born from communicating across two entirely different physical modalities.
Key Foundational Work
A Prototype Text to British Sign Language (BSL) Translation System (2003) — contributed 2 concept elements to this direction.
A Spanish speech to sign language translation system for assisting deaf-mute people (2006) — contributed 1 concept elements to this direction.