A Computational Approach to Sign Language Understanding

Tejaswini AnanthanarayanaAAAI 2021 · DOI

Abstract

Sign Language Translation (SLT) presents unique challenges due to its context-dependent nature and multi-channel delivery. This study investigates the efficacy of multi-modal pose features—incorporating body, hand, and facial data—within sequence-to-sequence models. Our findings demonstrate that a holistic approach to feature extraction is vital for bridging the communication gap between the Deaf and Hard-of-Hearing (DHH) community and the hearing population.

Introduction

Sign language is far more than a simple word-for-sign substitution of spoken language. It is a rich, multi-channel communication form that integrates hand shapes, movement, body posture, and facial expressions. Consequently, Automatic Sign Language Translation (SLT) faces significant hurdles. Primarily, the field struggles with contextual ambiguity, where signs are interpreted differently based on context or signer variation. Furthermore, identifying which visual features are most critical for accurate machine translation remains a complex feature complexity problem.

Methodology

The research employs a hybrid approach combining computer vision and natural language processing:

  1. Feature Extraction: Using OpenPose, the system extracts 2D coordinates for 25 body joints, 21 points per hand, and 70 face points. These are normalized into a "canonical form" to account for variations in camera distance and signer position.
  2. Deep Learning Models: The study evaluates Sequence-to-Sequence (Seq2Seq) models and Attention-rich Transformers to map visual sequences to spoken language text.
  3. Ablation Studies: To isolate the impact of specific inputs, the model was tested using various combinations of features (Hands, Body, Face).
  4. Human Oracle Experiment: Human ASL signers translated both raw RGB video and "skeleton-only" videos to quantify information loss during the transition from pixels to coordinates.

Key Insights: The Agreement Problem

A significant finding of this research is the inherent subjectivity in sign language interpretation. Even among human experts, "ground truth" is often elusive.

Comparison of ground truth captions and human signer interpretations showing the variability in translation.

As shown in the comparison above, interpretations of the same signed sequence can vary significantly between signers and the provided ground truth. This suggests that standard metrics like BLEU, which rely on exact word overlap, may underestimate model performance by penalizing valid but non-identical translations.

Results and Feature Contribution

The ablation studies confirm that holistic input is non-negotiable for high-quality translation.

Ablation study results showing the impact of different OpenPose feature combinations on BLEU scores across validation and test sets.

As detailed in the results table above, the "Hands + Body + Face" configuration achieved the highest performance, with a BLEU-1 score of 23.49 and a BLEU-4 of 5.55 on the test set. In contrast, using "Hands" alone yielded only 12.14 BLEU-1, and "Face" alone was nearly ineffective (2.23 BLEU-1). This confirms that while facial expressions provide essential context, they must be integrated with manual signs to convey specific meaning.

Discussion: The Skeleton Gap

The human oracle experiment highlighted a "Skeleton Gap": humans performed significantly worse when interpreting skeleton animations (BLEU-1: 6.32) compared to raw RGB video (BLEU-1: 24.86). This suggests that current pose-estimation models like OpenPose may discard subtle but vital information, such as mouthings or fine-grained finger movements, which are crucial for human understanding.

Conclusion and Future Work

While this study advances multi-modal SLT, limitations remain. The research was primarily conducted on the domain-specific RWTH-PHOENIX-Weather 2014 T dataset. Future work will explore optical flow to better capture movement dynamics and investigate the temporal importance of specific features—for instance, determining when facial features become the primary drivers of meaning during "question-asking" signs.

Summary updated

BibTeX
@inproceedings{ananthanarayana2021computational,
  author = {Tejaswini Ananthanarayana},
  title = {{A Computational Approach to Sign Language Understanding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17851},
}
Figure