Sign Language Translation with Hierarchical Spatio-TemporalGraph Neural Network

Jichao Kan, Kun Hu, Markus Hagenbuchner, Ah Chung Tsoi, Mohammed Bennamoun, Zhiyong Wang

Abstract

Sign language translation (SLT), which generates text in a spoken language from visual content in a sign language, is important to assist the hard-of-hearing community for their communications. Inspired by neural machine translation (NMT), most existing SLT studies adopted a general sequence to sequence learning strategy. However, SLT is significantly different from general NMT tasks since sign languages convey messages through multiple visual-manual aspects. Therefore, in this paper, these unique characteristics of sign languages are formulated as hierarchical spatio-temporal graph representations, including high-level and fine-level graphs of which a vertex characterizes a specified body part and an edge represents their interactions. Particularly, high-level graphs represent the patterns in the regions such as hands and face, and fine-level graphs consider the joints of hands and landmarks of facial regions. To learn these graph patterns, a novel deep learning architecture, namely hierarchical spatio-temporal graph neural network (HST-GNN), is proposed. Graph convolutions and graph self-attentions with neighborhood context are proposed to characterize both the local and the global graph properties. Experimental results on benchmark datasets demonstrated the effectiveness of the proposed method.

Introduction

Sign Language Translation (SLT) is significantly more complex than standard Neural Machine Translation (NMT). While NMT typically deals with text-to-text mapping, SLT must interpret "visual-manual" modalities: hand shapes, facial expressions, and body postures. Most existing methods treat SLT as a general sequence-to-sequence problem, often missing the nuanced interactions between different body parts. For example, a specific hand gesture might mean something different depending on the signer's facial expression.

To address this, researchers have developed a novel architecture called HST-GNN that treats sign language translation as a hierarchical graph problem. By modeling both high-level body regions (hands, face) and fine-grained skeletal joints across time, the system captures the multi-level, spatio-temporal relationships necessary for accurate translation.

Methodology

The Hierarchical Spatio-Temporal Graph Neural Network (HST-GNN) utilizes an encoder-decoder framework designed to mirror human perception of motion:

  1. Hierarchical Graph Construction: The encoder builds two levels of graphs. Level 1 (High-level) vertices represent major regions like the face and hands. Level 2 (Fine-level) vertices represent specific skeletal joints and facial landmarks (21 joints for hands, 29 landmarks for the face).
  2. Spatio-Temporal Modeling: To capture movement, the graphs include vertices from neighboring frames within a sliding temporal window. This allows the model to learn how joints move relative to each other (local) and how major body regions interact (global) over time.
  3. Graph Convolution & Transformer: Graph Convolutions (GCNs) capture local properties, while a novel Graph Transformer with self-attention captures global properties. A "neighborhood context" shortcut ensures global attention is guided by local spatial relationships.
  4. Two-Stage Decoder: The visual features are first converted into "glosses" (written words representing specific signs) using an LSTM and Connectionist Temporal Classification (CTC) loss. A second LSTM with an attention mechanism then translates this gloss sequence into natural spoken language text.
Overview of the HST-GNN architecture, featuring hierarchical graph encoding and a two-stage decoder.

Experimental Setup and Insights

The model was evaluated on three major benchmarks: PHOENIX-2014, PHOENIX-2014-T (German weather forecasts), and Chinese Sign Language (CSL). Several key insights emerged from the experiments:

  • Hierarchy is Essential: Modeling the body as a hierarchy (joints → regions → full pose) allows the model to extract features at the appropriate scale.
  • Local + Global Context: Combining GCNs with a Transformer architecture allows the model to understand context, such as distinguishing the same hand shape when moved near the chin versus the chest.
  • The "Goldilocks" Window: A temporal window size of 3 frames was found to be optimal. Smaller windows miss motion cues, while larger ones introduce noise.
  • Unsupervised Adjacency: Rather than using a fixed skeletal map, the model learns the relationships between joints in an unsupervised way, discovering non-obvious correlations.

Key Results

The HST-GNN achieved state-of-the-art (SOTA) performance across all benchmarks. On the PHOENIX-2014-T dataset, it achieved a Word Error Rate (WER) of 19.5 on the test set, significantly outperforming previous SOTA models like STMC (21.1). Translation quality, measured by the BLEU-4 metric, reached 22.3 on PHOENIX-2014-T and 17.8 on CSL.

Comparison of Word Error Rate (WER) across different models on Phoenix and CSL datasets.

Ablation studies confirmed that removing either the spatial graph or the temporal graph components led to significant performance drops, proving that the integrated spatio-temporal approach is critical to the model's success.

Conclusion

HST-GNN represents a significant step forward in automated sign language translation by explicitly modeling the hierarchical and spatio-temporal nature of sign language. While the model currently relies on pre-extracted skeletons, which can be sensitive to motion blur or occlusion, its superior performance on benchmarks validates the hierarchical graph approach. Future work will explore the use of 3D skeletal data and more complex directed message passing to further refine translation accuracy.

Summary updated

Abstract

Sign language translation (SLT), which generates text in a spoken language from visual content in a sign language, is important to assist the hard-of-hearing community for their communications. Inspired by neural machine translation (NMT), most existing SLT studies adopted a general sequence to sequence learning strategy. However, SLT is significantly different from general NMT tasks since sign languages convey messages through multiple visual-manual aspects. Therefore, in this paper, these unique characteristics of sign languages are formulated as hierarchical spatio-temporal graph representations, including high-level and fine-level graphs of which a vertex characterizes a specified body part and an edge represents their interactions. Particularly, high-level graphs represent the patterns in the regions such as hands and face, and fine-level graphs consider the joints of hands and landmarks of facial regions. To learn these graph patterns, a novel deep learning architecture, namely hierarchical spatio-temporal graph neural network (HST-GNN), is proposed. Graph convolutions and graph self-attentions with neighborhood context are proposed to characterize both the local and the global graph properties. Experimental results on benchmark datasets demonstrated the effectiveness of the proposed method.
BibTeX
@inproceedings{kan2022sign,
  author = {Jichao Kan and Kun Hu and Markus Hagenbuchner and Ah Chung Tsoi and Mohammed Bennamoun and Zhiyong Wang},
  title = {{Sign Language Translation with Hierarchical Spatio-TemporalGraph Neural Network}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00219},
}
Figure