Sign Language Production with Latent Motion Transformer

Pan Xie, Taiying Peng, Yao Du, Qipeng Zhang

Abstract

Sign Language Production (SLP) aims to generate realistic sign language videos from text or glosses. Traditional methods often rely on intermediate human pose estimations, which frequently fail to capture the rapid and nuanced hand movements essential for clear communication. We introduce the Latent Motion Transformer (LMT), a novel two-stage framework that bypasses pose estimation by modeling sign language directly in a compressed latent space. By combining a 3D VQ-GAN with a "perceptually-aware" autoregressive model, LMT achieves state-of-the-art performance in both visual quality and semantic accuracy on major sign language benchmarks like WLASL and NMFs-CSL.

Introduction

The primary challenge in SLP is the "two-step" hurdle: generating human pose sequences (skeletons) and then rendering those poses into video. Pose estimators often introduce artifacts or jitter, especially during complex hand gestures. While direct video-to-video generation exists, it is computationally expensive and often results in blurry outputs. LMT addresses these issues by operating in a discrete latent space that preserves visual detail and temporal dynamics without the need for explicit skeletal modeling.

Methodology

The LMT framework is divided into two distinct stages: video compression and latent sequence generation.

Stage 1: 3D VQ-GAN with Motion Transformer

The first stage employs a 3D Vector Quantized Generative Adversarial Network (VQ-GAN) to compress high-dimensional video data into a compact grid of discrete latent codes. A key innovation here is the Motion Transformer, which utilizes "trajectory attention." Unlike standard axial attention that processes rows and columns independently, trajectory attention tracks the movement of specific points over time. This approach reduces computational complexity from quadratic to linear while more effectively capturing the "choreography" of sign language.

Overview of the two-stage Latent Motion Transformer (LMT) framework.

Stage 2: Latent Transformer with Visual Perception

In the second stage, an autoregressive Transformer predicts the sequence of latent codes based on a provided sign gloss and an initial frame. To improve upon standard cross-entropy training, LMT incorporates a Gumbel-softmax trick. This allows the model to decode predicted tokens back into the pixel space during training, enabling the calculation of perceptual and reconstruction losses. This "visual feedback" ensures that the predicted latent sequences are not just statistically probable but also visually coherent and semantically accurate.

Experimental Setup

The model was evaluated on two word-level datasets:

  • WLASL (Word-Level American Sign Language): Contains approximately 21,000 clips across 2,000 words, featuring significant diversity in video quality and backgrounds.
  • NMFs-CSL (Chinese Sign Language): A curated dataset of 1,067 words characterized by rapid and centrally located sign movements.

Results and Discussion

LMT demonstrates superior performance over existing methods such as VideoGPT and LVT across multiple metrics.

Visual Quality and Semantic Accuracy

On the WLASL dataset, LMT achieved a Frechet Video Distance (FVD) of 301.62, significantly outperforming VideoGPT's 381.09. Furthermore, when using a pre-trained Sign Language Recognition (SLR) model to evaluate the generated videos, LMT reached an accuracy of 43.0%, a substantial improvement over the ~30% accuracy of previous latent transformer approaches.

Qualitative comparison of LMT against state-of-the-art methods on the WLASL dataset for the word 'Banana'.

Key Insights

  • Bypassing Poses: Avoiding the pose-estimation "middleman" prevents common artifacts like jittery hands and lost finger detail.
  • Trajectory Attention: This mechanism is better suited for sign language than standard attention, as it directly models the paths of moving hands.
  • Visual Feedback: Training the second-stage Transformer with visual losses was critical for achieving high-fidelity results, ensuring the "brain" of the model can "see" its own predictions.

Conclusion and Future Work

The Latent Motion Transformer represents a significant step forward in Sign Language Production by effectively combining 3D video quantization with motion-aware transformers. While LMT produces high-quality isolated word videos, future work will focus on scaling the architecture to handle continuous sign language sentences and improving the efficiency of downsampling to preserve even finer-grained finger movements.

Summary updated

BibTeX
@inproceedings{xie2024sign,
  author = {Pan Xie and Taiying Peng and Yao Du and Qipeng Zhang},
  title = {{Sign Language Production with Latent Motion Transformer}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00300},
}
Figure