Intermediate Representations for Avatar Sign Language Synthesis

23 concepts15 papers1 proposal

Translating spoken or written language into sign language is fundamentally different from translating between two spoken languages. Spoken languages are linear and sequential, relying on a single stream of audio. Sign languages, by contrast, are spatial, simultaneous, and multi-channel. A single sign utterance can concurrently combine hand shapes, trajectories, body orientations, and non-manual markers such as facial expressions and chin positions.

Because of this multi-channel complexity, mapping text directly to video or raw avatar joint movements is incredibly difficult and data-intensive. To overcome this, researchers have converged on a powerful architecture: decoupling the translation process using intermediate symbolic and phonetic representations. By breaking the problem into two distinct steps—translating text into a structured notation, and then using that notation to script a 3D avatar—systems achieve the precision, editability, and grammatical accuracy required for natural communication.

The Core Architecture: Decoupling Translation from Rendering

The primary reason direct text-to-avatar translation fails in complex domains is the structural mismatch between the input and output modalities. As Mathias Müller et al. point out in their overview of the First WMT Shared Task on Sign Language Translation (WMT-SLT22), sign language production requires modeling intricate spatial grammar and non-manual markers that do not map one-to-one with written words.

To bypass this bottleneck, modern synthesis pipelines employ a two-stage pipeline:

$$\text{Source Text/Speech} \xrightarrow{\text{Stage 1: Translation}} \text{Intermediate Representation} \xrightarrow{\text{Stage 2: Synthesis}} \text{3D Avatar Animation}$$

As Catarina Sousa et al. demonstrate in Enhancing Accessible Communication: from European Portuguese to Portuguese Sign Language, utilizing an intermediate symbolic representation like glosses provides a critical linguistic mapping layer. This abstraction avoids the extreme complexity of direct text-to-video generation, allowing the system to resolve grammatical differences before worrying about the physics of joint rotation.

[ Written Text ] 
       │
       ▼ (Linguistic Translation)
[ Intermediate Notation (Glosses / AZee / HamNoSys) ]
       │
       ▼ (Geometric / Phonetic Scripting)
[ 3D Avatar Animation Engine ]

Linguistic Abstraction: Glosses and Grammatical Frameworks

The first layer of intermediate representation is typically linguistic. Glosses—words from a spoken language written in capital letters to represent individual signs—serve as a baseline semantic bridge. For example, the system built by Rubén San-Segundo et al. in Advanced speech communication system for deaf people utilizes a visual interface that allows users to organize sign language sequences using glosses, providing a structured way to define dates, times, and specific terms before rendering.

However, glosses alone cannot capture the spatial syntax or simultaneous non-manual markers of sign language. To solve this, researchers have developed formal geometric and grammatical frameworks:

  • The AZee Framework: To capture conversational nuances, Emmanuella Martinod et al. leverage the AZee framework in Formal Representation of Interrogation in French Sign Language. AZee formalizes sign language grammar by mapping meaning directly to specific physical constraints, such as proving that chin advancement is a more reliable marker for interrogation in French Sign Language than eyebrow movement.
  • The AZVD System: Moving toward a visual syntax, Michael Filhol et al. introduced AZVD in A software editor for the AZVD graphical Sign Language representation system. This system uses a formal graphical language of 2D symbols that can be recursively combined to represent complex utterances. Because the system defines strict production rules linking these symbols to articulated forms, any diagram constructed in the editor can be automatically synthesized into avatar movements.

Phonetic and Geometric Control: Driving the Avatar

Once the linguistic structure is defined, it must be translated into physical instructions for the avatar. This is achieved by converting linguistic abstractions into phonetic notations or markup languages that specify hand shapes, locations, and movements.

Early foundational work by Ian Marshall et al. in A Prototype Text to British Sign Language (BSL) Translation System established the utility of converting BSL-oriented representations into phonetic descriptions to drive virtual human animations. To make these phonetic descriptions machine-readable and scalable, Manny Rayner et al. in An Open Web Platform for Rule-Based Speech-to-Sign Translation encoded phonetic notations into an XML-based markup language. This XML layer acts as the final bridge, translating abstract glosses into the exact joint coordinates required for 3D rendering.

[ Gloss / Grammar Layer ] ──> [ Phonetic Notation (HamNoSys) ] ──> [ XML Markup ] ──> [ Joint Coordinates ]

This phonetic approach provides unparalleled editability. Tsubasa Uchida et al. demonstrated this in HamNoSys-based Motion Editing Method for Sign Language by using the Hamburg Notation System (HamNoSys) to modify existing motion-capture data. Instead of recording new sessions, developers can alter the HamNoSys code to adjust specific linguistic components, such as classifier predicates, significantly improving avatar intelligibility. Similarly, the SAILSE system developed by V. López-Ludeña et al. (Developing an information system for deaf) relies on a custom editor supporting multiple notation systems (glosses, SEA, and HamNoSys) to facilitate detailed, multi-layered sign composition.

Why the Intermediate Approach Endures

The ultimate validation of intermediate representations lies in their practical deployment and legibility. Early systems, such as the Spanish speech-to-sign translator built by R. San-Segundo et al. (A Spanish speech to sign language translation system for assisting deaf-mute people) and the cloud-based system by Javier Tejedor et al. (An on-line, cloud-based Spanish-Spanish sign language translation system), proved that database-driven avatar animations could successfully deliver critical public services in real-time. More recently, Nicolas Tagliabue et al. demonstrated in DEEP: an automatic bidirectional translator leveraging an ASR for translation of Italian sign language that synthetic avatar generation can achieve a clarity rate of 98.2% in domain-specific environments like pharmacies.

Even as the field transitions toward machine learning and large language models, structured intermediate representations remain indispensable. In SignLLM: Sign Language Production Large Language Models, Sen Fang et al. show that enriching the generation pipeline with language-specific attributes allows LLMs to accurately map natural language queries to precise sign gestures. Furthermore, John McDonald et al. demonstrate in Multilingual Synthesis of Depictions through Structured Descriptions of Sign: An Initial Case Study that structured linguistic descriptions can capture geometric constructions and timing rules common to multiple sign languages.

While simplified models like stick figures are useful for rapid algorithmic evaluation (Alfarabi Imashev et al., Comparative Analysis of Sign Language Interpreting Agents Perception: A Study of the Deaf), the path to truly natural, cross-lingual sign language production relies on these rich, intermediate symbolic layers. They turn the chaotic, multi-channel physical reality of signing into a structured, programmable language that digital avatars can reliably speak.

Key Foundational Work

A Prototype Text to British Sign Language (BSL) Translation System (2003) — contributed 1 concept elements to this direction.
A Spanish speech to sign language translation system for assisting deaf-mute people (2006) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_avatar_based_sign_language_synthesis_2239de43,
  title = {avatar-based sign language synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/avatar-based-sign-language-synthesis/rdr_2239de43f0924fad9ed50c48b022ae96},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_avatar_based_sign_language_synthesis_2239de43_v1,
  title = {avatar-based sign language synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/avatar-based-sign-language-synthesis/rdr_2239de43f0924fad9ed50c48b022ae96/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake