Abstract
Sign Language Translation (SLT) is a challenging task due to the multi-channel nature of sign languages, which involve complex interactions between manual articulations and non-manual signals. Traditional models often treat video frames as flat sequences, neglecting the inherent spatial structure of the human body. This paper introduces the Hierarchical Spatio-Temporal Graph Neural Network (HST-GNN), a novel framework that models sign language as a structured graph problem. By organizing body parts into a two-level hierarchy and capturing both local and global dependencies across time, HST-GNN achieves state-of-the-art performance on major benchmarks, including PHOENIX-2014-T and Chinese Sign Language (CSL) datasets.
Introduction
Sign Language Translation (SLT) extends beyond simple text-to-text conversion. It requires interpreting a "multi-channel" communication system where meaning is conveyed through hand shapes, movements, facial expressions, head tilts, and body posture. Most existing SLT models utilize standard convolutional neural networks (CNNs) to extract features from video frames, treating them as a flat sequence. However, this approach often fails to capture the intricate spatial relationships between different body parts—such as the specific interaction between a hand and the face—which are crucial for disambiguating signs.
To address this, we propose the Hierarchical Spatio-Temporal Graph Neural Network (HST-GNN). This model explicitly represents the signer's body as a hierarchical graph, allowing for the fine-grained modeling of both local joint movements and global body-part interactions over time.
Methodology
The HST-GNN framework consists of a hierarchical graph encoder followed by a two-stage decoder.
Hierarchical Graph Construction
The signer's body is organized into a two-level hierarchy:
- High-level (Level 1): Vertices represent major body regions, including the face, left hand, and right hand. Edges model the interactions between these regions.
- Fine-level (Level 2): Vertices represent specific landmarks, such as 29 facial points and 21 hand joints, capturing the detailed "shape" of each sign.
To incorporate temporal dynamics, the model employs a sliding window of frames, connecting vertices across time to form a 3D spatio-temporal graph structure.
HST-GNN Encoder
The encoder utilizes a combination of Graph Convolutional Networks (GCNs) and Graph Transformers:
- GCNs are used to extract local patterns, focusing on the relationships between a joint and its immediate neighbors.
- Graph Transformers leverage self-attention to capture global properties, enabling the model to understand long-range dependencies between distant parts of the graph, such as the coordination between a hand gesture and a specific facial expression.
- Hierarchical Pooling then fuses fine-level joint data into high-level body-part representations, resulting in a comprehensive latent vector for the video sequence.
Two-Stage Decoder
The decoding process is split into two stages:
- Gloss Generation (feats2gloss): Visual features are converted into "glosses" (written labels for signs) using a Connectionist Temporal Classification (CTC) loss.
- Translation (gloss2text): The sequence of glosses is translated into a natural spoken language sentence using an attention-based LSTM.
Experiments and Results
The model was evaluated on three major benchmarks: PHOENIX-2014, PHOENIX-2014-T (German weather forecasts), and Chinese Sign Language (CSL).
Key Insights
- Hierarchy Matters: Explicitly modeling the hierarchy of the human body significantly outperforms "flat" neural network architectures. A finger's movement is most meaningful when contextualized within the hand, which in turn is contextualized within the whole body.
- Spatio-Temporal Balance: The study found that a temporal window of 3 frames provided the optimal balance between capturing movement and avoiding noise.
- Local-Global Synergy: Combining GCNs for local details and Transformers for global context prevents the model from missing critical fine-grained information or losing the "big picture."
Performance Comparison
HST-GNN achieved state-of-the-art results across all evaluated datasets. On the PHOENIX-2014-T test set, it reached a Word Error Rate (WER) of 19.5, a significant improvement over previous models like STMC (21.1).
Qualitative analysis further revealed that the graph-based representation is particularly effective at reducing "insertion errors"—hallucinated words—by providing a more physically grounded representation of the signer's movements.
Conclusion and Future Work
The HST-GNN demonstrates the power of hierarchical graph modeling for Sign Language Translation. By treating the human body as a structured entity rather than a flat feature vector, the model achieves superior translation quality.
However, the current approach depends on external skeleton extraction tools (e.g., OpenPose), which can be sensitive to motion blur or occlusions. Future research will focus on developing end-to-end graph construction methods to reduce this dependency and exploring directed graph edges to better model the directional "flow" of sign language gestures.
Summary updated