Word-Conditioned 3D American Sign Language Motion Generation

Lu Dong, Xiao Wang, Ifeoma NwoguACL 2024 · OpenReview

Abstract Sign words are the building blocks of any sign language. In this work, we present wSignGen, a word-conditioned 3D American Sign Language (ASL) generation model dedicated to synthesizing realistic and grammatically accurate motion sequences for sign words. Our approach leverages a transformer-based diffusion model, trained on a curated dataset of 3D motion meshes from word-level ASL videos. By integrating CLIP, wSignGen offers two advantages: image-based generation, which is particularly useful for children learning sign language but not yet able to read, and the ability to generalize to unseen synonyms. Experiments demonstrate that wSignGen significantly outperforms the baseline model in the task of sign word generation. Moreover, human evaluation experiments show that wSignGen can generate high-quality, grammatically correct ASL signs effectively conveyed through 3D avatars.

Introduction

Sign Language Production (SLP)—the task of generating sign language from text or other modalities—remains a significant challenge in AI research. Most existing systems are "closed-set," restricted to producing only the specific signs seen during training. For example, a system trained on "table" might fail to produce a sign for "desk" despite their semantic proximity. Furthermore, ASL requires precise coordination of manual cues (handshape, movement) and non-manual cues (facial expressions), making natural 3D synthesis difficult.

We introduce wSignGen, a model designed to generate realistic 3D ASL animations from text or image inputs. By leveraging a transformer-based diffusion model and CLIP embeddings, wSignGen achieves semantic generalization and supports image-to-sign translation, opening new possibilities for accessibility and early childhood education for the Deaf and Hard-of-Hearing (DHH) community.

Methodology

The wSignGen architecture departs from traditional Variational Autoencoders (VAEs) in favor of a Diffusion Model. Key components include:

  • CLIP Integration: By using CLIP (Contrastive Language-Image Pre-training) embeddings, the model maps text and images into a shared semantic space. This allows the model to handle synonyms and generate signs directly from visual prompts.
  • Transformer-based Diffusion: The model learns to denoise random motion into structured ASL sequences by predicting the motion signal ($X_0$) directly at each step.
  • Velocity Loss: To ensure smooth, natural motion and eliminate jitter, a specialized loss function penalizes unrealistic speed changes between frames.
  • SMPLX Body Model: Outputs are mapped to the SMPLX format, providing detailed 3D representations of hand and finger movements.

Key Insights: Semantic Generalization and Image-to-Sign

A core innovation of wSignGen is its ability to understand the meaning behind a word rather than just learning a fixed mapping. Because of the CLIP integration, the model can generate signs for unseen words if they are semantically related to training data (e.g., generating "desk" based on "table"). Additionally, the model can "translate" images into signs, a feature specifically designed for DHH children who may not yet be literate in written language.

wSignGen demonstrates semantic generalization to unseen synonyms (top) and image-to-sign generation for literacy support (bottom).

Experiments and Results

The model was trained on a curated subset of the WLASL dataset, featuring the 103 most frequent signs with 3D features extracted via the Hand4whole model. Evaluation was conducted on a specialized ASL3D$_S$ subset.

Quantitative results show that a pre-trained sign classifier identified wSignGen’s motions with 100% accuracy, outperforming VAE-based baselines. In image-based tests using external web images, the model achieved 86% accuracy. Human evaluations, including feedback from DHH individuals, gave the model an average score of 8.36/10, praising the accuracy of handshapes and spatial positioning.

Examples of high-quality 3D ASL motion sequences generated by wSignGen for the words "TABLE" and "MANY".

Discussion and Future Work

While wSignGen represents a leap forward, limitations remain. The current vocabulary is limited to 103 words, and the model focuses primarily on manual cues, omitting vital non-manual features like facial expressions. Future research will aim to scale the vocabulary, incorporate full-body grammar including facial mouthing, and transition from single-word generation to continuous ASL sentences.

Summary updated

Abstract

Sign words are the building blocks of any sign language. In this work, we present wSignGen, a word-conditioned 3D American Sign Language (ASL) generation model dedicated to synthesizing realistic and grammatically accurate motion sequences for sign words. Our approach leverages a transformer-based diffusion model, trained on a curated dataset of 3D motion meshes from word-level ASL videos. By integrating CLIP, wSignGen offers two advantages: image-based generation, which is particularly useful for children learning sign language but not yet able to read, and the ability to generalize to unseen synonyms. Experiments demonstrate that wSignGen significantly outperforms the baseline model in the task of sign word generation. Moreover, human evaluation experiments show that wSignGen can generate high-quality, grammatically correct ASL signs effectively conveyed through 3D avatars.
BibTeX
@inproceedings{dong2024word,
  author = {Lu Dong and Xiao Wang and Ifeoma Nwogu},
  title = {{Word-Conditioned 3D American Sign Language Motion Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=ssItQUwei7},
}
Figure