Modern natural language processing (NLP) operates on a fundamental assumption: language can be broken down into a sequence of discrete, standardized textual tokens. Spoken languages map neatly to written scripts, and their acoustic profiles can be segmented into well-defined phonetic subunits.
Sign languages break this assumption entirely. They are visual-gestural languages that utilize three-dimensional space, facial expressions, and body posture simultaneously. Crucially, they lack a universally accepted, standardized written form. This absence of a textual baseline creates a severe digital divide, excluding signed languages from traditional NLP pipelines and creating significant information accessibility barriers.
To bridge this gap, researchers are developing computational representations that formalize the visual-gestural modality into structures that machines can parse, index, and translate.
Why Text Captions Are Not Enough
A common misconception is that text-based tools like closed captions or subtitles solve digital accessibility for the Deaf and Hard of Hearing community. However, as Sudha Krishnamurthy et al. point out, captions are often insufficient for individuals whose primary or native language is sign language, particularly children or those with limited literacy in the dominant spoken language.
Sign languages are not word-for-word visual encodings of spoken languages. As Kazuko Takahashi demonstrates, true sign languages possess entirely independent grammars, spatial structures, and syntax. For instance, a native signer interacting with public services may struggle with written text due to these profound conceptual and grammatical differences, as noted by V. López-Ludeña et al. and Javier Tejedor et al.. Because video files are heavy and inherently difficult to edit, search, or index, the field requires a discrete intermediate representation to make sign language computationally tractable.
Designing a Script for the Unwritten
To apply machine translation and NLP techniques, researchers must map continuous video data to a structured intermediate format. One prominent approach is the use of specialized notation systems.
[Spoken/Written Text] <---> [Intermediate Structural Script] <---> [3D Avatar / Video Generation]
(SignWriting, Glosses, AZVD)
For example, Zifan Jiang et al. utilized SignWriting—a visual script that factorizes signs into geometric symbols representing handshapes, movements, and locations—as an intermediate step in a machine translation pipeline. By parsing and factorizing this script, the authors successfully adapted traditional text-based translation models to process signed content.
Another common approach is "gloss notation," where sign sequences are transcribed using capitalized words from a spoken language to represent distinct concepts. However, glossing introduces its own bottlenecks. Rubén San-Segundo et al. observe that the lack of complete standardization in gloss notation introduces ambiguity, which complicates both the translation process and the user's ability to input signs consistently. To address the limitations of existing systems, Michael Filhol et al. introduced AZVD, a graphical representation designed to be both formal enough for computational avatar synthesis and intuitive enough for adoption by the signing community.
The Subunit Granularity Problem
In speech recognition, audio signals are mapped to a finite set of phonemes. Sign language recognition lacks a universal set of standardized subunits, making it difficult to determine how granular an algorithmic model should be.
To look up an unfamiliar sign in a dictionary, a user cannot rely on an alphabetical index. Helen Cooper et al. addressed this by breaking signs down into coarse linguistic sub-units—specifically motion, location, and handshape—allowing a system to classify and retrieve signs directly from video queries.
When modeling sign language sequentially, choosing the right number of states for a recognition model is notoriously difficult. Sandrine Tornay et al. tackled this problem by developing a Hidden Markov Model (HMM) framework with inherent model selection. Instead of forcing a rigid, predefined phonetic structure onto a sign, their system dynamically determines the appropriate structural complexity during the recognition process, leading to more robust modeling across different sign languages.
Lexical Gaps and Technical Vocabulary
The challenge of standardization extends beyond structural representation into the lexicon itself. Many sign languages suffer from a scarcity of standardized technical vocabulary, particularly in STEM fields.
When a standardized conceptual sign does not exist for a complex scientific term, signers are forced to rely on letter-by-letter fingerspelling. Kayo Yin et al. highlighted this bottleneck by introducing the ASL STEM Wiki dataset. They demonstrated that the lack of standardized vocabulary creates a linguistic hurdle for technical education, underscoring the need for AI tools that can not only recognize fingerspelling but also help suggest and propagate new, standardized conceptual signs within the community.
By moving away from raw video processing and toward structured, intermediate computational representations, researchers are laying the groundwork for sign languages to become first-class citizens in the modern digital landscape.
Key Foundational Work
Project for production of closed-caption TV programs for the hearing impaired (1998) — contributed 1 concept elements to this direction.
A Chart-Parsing Algorithm for Efficient Semantic Analysis (2002) — contributed 1 concept elements to this direction.