Non-Autoregressive Sign Language Production via Knowledge Distillation

Eui Jun Hwang, Jung Ho Kim, Suk Min Cho, Jong C. ParkarXiv 2022 · arXiv

TL;DR

Researchers from KAIST have developed a novel Non-Autoregressive (NAR) model for Sign Language Production that generates realistic skeletal pose sequences directly from text. By combining Knowledge Distillation from a pre-trained pose encoder with a dedicated length regulator, the model overcomes common pitfalls like error propagation and "false decoding initiation," setting new state-of-the-art benchmarks for realism and translation accuracy.

What Problem Does This Paper Address?

Sign Language Production (SLP) aims to translate spoken or written language into sign language, typically represented as a sequence of skeletal poses. Historically, this has been approached in two ways:

  1. AutoRegressive (AR) models: These generate poses one frame at a time, using the previous frame to predict the next. This often leads to "regression to the mean" (stiff, average-looking poses) and "error propagation," where a small mistake in one frame snowballs into a nonsensical sequence.
  2. Non-Autoregressive (NAR) models: These generate the entire sequence in parallel, which is faster and avoids error propagation. However, existing NAR models struggle with "false decoding initiation"—where the beginning of a sequence is poorly formed—and often lack a mechanism to predict exactly how long a sign sequence should be.

This paper tackles these NAR limitations to produce more fluid, detailed, and correctly-timed sign language.

Methods

The authors propose a framework consisting of three main components, illustrated in Figure 2:

Fig. 2: An overview of the proposed NAR-SLP model. To apply Knowledge Distilla- tion (KD), we pre-trained a sign pose network based on VAE. The trained encoder and decoder are used as Latent Guiding Module (LGM) and Pose Generating Mod- ule (PGM), respectively. Latent Learning Module (LLM) generates a length-regulated sequence of latent representation. ⊕denotes a length regulating operation. LLM con- sists of three sub-modules: BERT as a language embedding module, Transformer-based NAR decoder, and Length Regulator. During training, as a pose distillation process, LGM acts as a teacher network, teaching LLM to generate zlang similar to zpose.
  • Latent Guiding Module (LGM - The Teacher): A pre-trained Variational Autoencoder (VAE) encoder that has learned to map real sign poses into a rich latent space ($z_{pose}$).
  • Latent Learning Module (LLM - The Student): This is the core of the new model. It uses BERT to encode the input text and a NAR Transformer Decoder to generate a sequence of latent vectors ($z_{lang}$). Crucially, it includes a Length Regulator that uses the BERT [CLS] token to predict the total duration of the sign sequence.
  • Pose Generating Module (PGM): The pre-trained VAE decoder that transforms the latent vectors back into actual 3D skeletal poses.

Key Innovations: * Knowledge Distillation (KD): During training, the "Student" (LLM) is forced to produce latent vectors that match the "Teacher's" (LGM) vectors. This aligns linguistic features directly with spatial pose features, effectively solving the "false decoding initiation" problem. * Binary Cross Entropy (BCE) Loss: Instead of the standard Mean Squared Error (MSE), the authors optimize their VAE using BCE. They treat pose coordinates as probabilities in a multivariate Bernoulli distribution, which helps the model capture fine-grained details like hand movements that MSE often "blurs" out.

Datasets

The model was trained and evaluated on the RWTH-PHOENIX-Weather 2014T dataset. This is the gold standard in the field, containing over 8,000 pairs of German spoken language and German Sign Language (DGS) videos, specifically from weather forecasts. Poses were extracted using OpenPose and lifted into 3D.

Key Insights and Ideas

  • KD as an Alignment Tool: In this context, Knowledge Distillation isn't just for model compression. It acts as a bridge between two very different modalities: the discrete world of text and the continuous, high-dimensional world of human motion.
  • Parallelism Wins for Long Sequences: Because the model is non-autoregressive, its performance doesn't degrade as the sign sequence gets longer. AR models tend to "drift" or become static during long sentences, whereas this model maintains consistent quality.
  • Text-to-Pose (T2P) > Gloss-to-Pose (G2P): The authors reinforce the idea that translating directly from text is superior to using intermediate "glosses" (simplified sign labels), as glosses often strip away rich linguistic information necessary for natural movement.
Fig. 1: An overview of Sign Language Production (SLP). There are two different tasks: Gloss-to-Pose (G2P) and Text-to-Pose (T2P). Recent SLP works demonstrate that T2P performs better than G2P. Hence, we focus on the T2P task in this work.

Key Results

The proposed model significantly outperformed previous state-of-the-art (SOTA) models like Progressive Transformers (PT) and NSLP-G across multiple metrics:

  • Fréchet Gesture Distance (FGD): Achieved a score of 0.15 (lower is better), compared to 0.22 for the previous best NAR model and 0.33 for the best AR model. This indicates the generated poses are much closer to the distribution of real human signing.
  • Back-Translation (BT): When the generated poses were translated back into text by a separate pre-trained model, the results achieved a BLEU-4 score of 13.90 and a ROUGE score of 36.89, both new SOTA marks.
  • Qualitative Improvement: Visual inspections showed that the model produces much more detailed hand expressions and starts sequences with more accurate poses compared to its predecessors.

Limitations and Future Work

  • Back-Translation Reliability: The authors noted a strange phenomenon where their model occasionally scored higher than the ground truth in back-translation. This suggests that BT metrics are heavily dependent on the quality of the translation model used for evaluation and may not be an absolute measure of sign quality.
  • Spatial Modeling: While the model handles temporal aspects well, the authors suggest that future work could incorporate Graph Neural Networks to better model the physical constraints and relationships between human joints.
  • Dataset Diversity: The study was limited to the PHOENIX weather dataset. The authors plan to apply their model to newer, larger datasets like "How2Sign" to test its generalizability.

Summary updated

Abstract

Sign Language Production (SLP) aims to translate expressions in spoken language into corresponding ones in sign language, such as skeleton-based sign poses or videos. Existing SLP models are either AutoRegressive (AR) or Non-Autoregressive (NAR). However, AR-SLP models suffer from regression to the mean and error propagation during decoding. NSLP-G, a NAR-based model, resolves these issues to some extent but engenders other problems. For example, it does not consider target sign lengths and suffers from false decoding initiation. We propose a novel NAR-SLP model via Knowledge Distillation (KD) to address these problems. First, we devise a length regulator to predict the end of the generated sign pose sequence. We then adopt KD, which distills spatial-linguistic features from a pre-trained pose encoder to alleviate false decoding initiation. Extensive experiments show that the proposed approach significantly outperforms existing SLP models in both Frechet Gesture Distance and Back-Translation evaluation.
BibTeX
@misc{hwang2022non,
  author = {Eui Jun Hwang and Jung Ho Kim and Suk Min Cho and Jong C. Park},
  title = {{Non-Autoregressive Sign Language Production via Knowledge Distillation}},
  year = {2022},
  eprint = {2208.06183},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.06183},
}
Figure