Sign Stitching: A Novel Approach to Sign Language Production

Harry Walsh, Ben Saunders, Richard Bowden

Introduction

Sign Language Production (SLP)—the automated translation of spoken or written text into sign language video—faces significant challenges in generating expressive and natural movements. Traditional deep learning approaches often suffer from "regression to the mean," where models trained on varied datasets produce "average" poses that are under-articulated, blurry, or robotic. Furthermore, these systems frequently neglect non-manual features like facial expressions and prosody (rhythm and stress), which are essential for clear communication in the Deaf community.

To address these issues, researchers have introduced "Sign Stitching," a novel pipeline that shifts the paradigm from generating poses from scratch to intelligently joining high-quality dictionary examples. This approach achieves state-of-the-art performance by preserving the articulation of human-recorded signs while ensuring smooth, natural transitions.

The Sign Stitching Pipeline

The proposed system decomposes the SLP task into three primary modules: a translation model, a sophisticated stitching engine, and a photo-realistic renderer.

Overview of the Sign Stitching pipeline, showing the progression from spoken language text to the final sign video.
  1. Translation Model: An encoder-decoder Transformer architecture translates spoken text into a sequence of glosses (sign labels). Crucially, it also predicts sign durations, facial expression tokens, and a low-pass filter "cutoff" frequency that dictates the stylistic "vibe" of the sequence.
  2. 7-Step Stitching Module: This core component transforms the predicted gloss sequence into a continuous 3D skeleton pose. It involves selecting signs from a dictionary, normalizing them to a canonical pose, cropping rest frames, and resampling them to match predicted rhythms.
  3. Smart Stitching & Filtering: Transitions are calculated based on hand velocity, and a low-pass Butterworth filter is applied. This filter smooths transitions and allows the model to emulate different signing styles (e.g., "soft" vs. "fast") by adjusting the frequency cutoff.
  4. Facial Expression & Rendering: Realistic facial movements are generated via a Noise Substitution Vector Quantization (NSVQ) architecture. Finally, SignGAN converts the 3D skeleton into a photo-realistic human video.
Detailed view of the 7-step stitching process used to create cohesive motion from isolated dictionary signs.

Evaluation and Results

The model was evaluated against major sign language benchmarks, including PHOENIX14T (German weather forecasts), mDGS (large-scale German discourse), and BSLCPT (British Sign Language).

The Sign Stitching approach demonstrated substantial improvements over existing state-of-the-art models:

  • Quantitative Performance: On the BSLCPT and mDGS datasets, the model achieved BLEU-1 score improvements of 269% and 98%, respectively, compared to the previous "Progressive Transformer" baseline.
  • Human Preference: In a user study with native and L2 signers, 87.5% of participants preferred the Sign Stitching output over the baseline.
  • Realism: 100% of surveyed users agreed that the frequency-domain filtering significantly enhanced the realism of the generated signing.

Discussion and Future Directions

The success of Sign Stitching highlights the importance of expressivity and prosody in SLP. By utilizing real, isolated signs, the system bypasses the articulation issues common in generative models. The use of a Butterworth filter provides an elegant method for controlling motion style, while duration resampling ensures the signing feels human rather than mechanical.

However, limitations remain. The system's performance is inherently tied to the quality and breadth of its sign dictionary. While "Smart Stitching" handles transitions effectively, it does not yet fully capture "co-articulation"—the fluid way human signers overlap the end of one sign with the beginning of the next. Future research will focus on bridging the remaining performance gap to ground-truth human signing and improving the modeling of complex co-articulation effects.

Summary updated

Abstract

Sign Language Production (SLP) is a challenging task, given the limited resources available and the inherent diversity within sign data. As a result, previous works have suffered from the problem of regression to the mean, leading to under-articulated and incomprehensible signing. In this paper, we propose using dictionary examples to create expressive sign language sequences. However, simply concatenating the signs would create robotic and unnatural sequences. Therefore, we present a 7-step approach to effectively stitch the signs together. First, by normalising each sign into a canonical pose, cropping and stitching we create a continuous sequence. Then by applying filtering in the frequency domain and resampling each sign we create cohesive natural sequences, that mimic the prosody found in the original data. We leverage the SignGAN model to map the output to a photo-realistic signer and present a complete Text-to-Sign (T2S) SLP pipeline. Our evaluation demonstrates the effectiveness of this approach, showcasing state-of-the-art performance across all datasets.
BibTeX
@inproceedings{walsh2024sign,
  author = {Harry Walsh and Ben Saunders and Richard Bowden},
  title = {{Sign Stitching: A Novel Approach to Sign Language Production}},
  booktitle = {BMVC},
  year = {2024},
}
Figure