Abstract
Sign Language Recognition (SLR) and Translation (SLT) systems typically process high-resolution RGB video, which is computationally expensive and sensitive to environmental noise. This paper introduces the Multi-Stream Keypoint Attention Network (MSKA), a novel framework that operates exclusively on skeletal keypoints. By decoupling keypoints into four specialized streams—left hand, right hand, face, and body—and utilizing a pure attention mechanism, MSKA captures the complex interplay between different articulators. The model achieves state-of-the-art results on the Phoenix-2014T benchmark, significantly narrowing the performance gap between keypoint-based and video-based methods.
Introduction
Most contemporary SLR and SLT models rely on 3D Convolutional Neural Networks (CNNs) to extract features from raw video frames. While effective, these "heavy" backbones are prone to overfitting on background details and require substantial hardware resources. Keypoint-based methods offer a robust alternative by focusing on the coordinates of joints and facial landmarks. However, previous keypoint models often used rigid graph structures that failed to capture the fluid, implicit relationships between different body parts (e.g., how a facial expression modifies a manual sign).
Methodology: The MSKA Architecture
The MSKA framework addresses the limitations of prior skeletal models through three primary innovations:
1. Keypoint Decoupling
Rather than treating all 133 keypoints as a single input vector, MSKA splits them into four distinct streams: Left Hand, Right Hand, Face, and Whole Body. This multi-stream approach mimics human perception, which processes manual gestures and non-manual markers (like mouth shapes) in parallel.
2. Pure Attention with Tanh Activation
Unlike traditional Graph Convolutional Networks (GCNs) that rely on predefined adjacency matrices, MSKA uses a pure self-attention mechanism to learn joint relationships dynamically. A key modification is the replacement of the standard Softmax activation with Tanh in the attention map. This allows the model to represent "negative correlations," learning that certain movements might inhibit or exclude others, leading to a more expressive representation of sign language.
3. Multi-Stream Fusion and Self-Distillation
To integrate information across streams, a "Fuse Head" aggregates features from all four branches. During training, the model employs frame-level self-distillation, where the ensemble's collective knowledge is used to supervise and refine the individual streams, improving the robustness of each branch.
Translation Extension
For the translation task (MSKA-SLT), the features extracted by the recognition model are passed through a Multi-Layer Perceptron (MLP) and fed into a pre-trained mBART (multilingual Bidirectional and Auto-Regressive Transformer). This allows the system to translate sequence-based sign features into coherent natural language sentences.
Key Results and Insights
The model was evaluated on major benchmarks including Phoenix-2014, Phoenix-2014T, and CSL-Daily.
- State-of-the-Art Performance: MSKA-SLT achieved a BLEU-4 score of 29.03 on the Phoenix-2014T test set, a record for keypoint-based translation.
- Closing the RGB Gap: MSKA reached a 20.5% Word Error Rate (WER) on Phoenix-2014T, outperforming several complex video-based architectures and proving that keypoints can be as effective as raw pixels when modeled correctly.
- Dominant Hand Importance: Ablation studies showed that the right-hand stream (typically the dominant hand for signers in these datasets) contributed significantly more to accuracy than the left-hand stream, with a 22% difference in error rates.
- Efficiency: By bypassing heavy video backbones, MSKA is significantly faster and more suitable for real-time deployment on edge devices.
Conclusion
The Multi-Stream Keypoint Attention Network demonstrates that sign language can be effectively recognized and translated using only skeletal data. By moving away from fixed graph topologies and embracing a decoupled, attention-driven architecture, MSKA provides a computationally efficient and robust solution for bridging the communication gap between the Deaf and hearing communities.
Summary updated