Making Body Movement in Sign Language Corpus Accessible for Linguists and Machines with Three-Dimensional Normalization of MediaPipe

Victor Skobov, Mayumi BonoEMNLP 2023 · DOI

Abstract

Sign language recognition and linguistic analysis often struggle with the variability of 2D video data. This paper introduces a novel 3D normalization technique for MediaPipe pose data, transforming raw coordinates into a standardized "joint tree" based on human proportions. This method not only achieves state-of-the-art accuracy in sign recognition but also creates human-interpretable visual encodings, revealing unique "movement fingerprints" in signers.

Introduction

Computer vision tools like MediaPipe are effective at tracking body parts in 2D, but they face significant hurdles in sign language applications. The primary issue is camera dependency: 2D coordinates shift drastically with camera angle changes, even if the sign remains identical. This hinders the generalization of AI models across different datasets. Furthermore, there is a linguistic gap; raw X, Y coordinates are unintelligible to linguists, who traditionally rely on manual, subjective annotation. This research proposes a middle ground that mathematically analyzes movement while maintaining human-readable context.

Methodology: 3D Normalization and Visual Encoding

The proposed solution involves a two-step process: 3D Normalization and RGB Representation.

3D Normalization via "Joint Trees"

Instead of treating body points as independent dots, the researchers model the body as a hierarchical tree rooted at the neck. By assuming fixed human proportions and using the inter-ocular distance as a baseline, the system estimates the missing depth (Z-axis) for all limbs. The resulting digital skeleton is then rotated and scaled to face a virtual camera, ensuring consistency regardless of the original camera's position.

Comparison of standard 2D normalization vs. the proposed 3D normalization across different camera views.

Direction Angle and Visual Encoding

To translate this 3D data into a usable format, the system calculates "direction angles"—the orientation of a limb relative to the X, Y, and Z axes. These angles are then mapped directly into the Red, Green, and Blue (RGB) channels of an image.

Mapping 3D direction angles to RGB color channels for visual encoding.

This "Visual Encoding" creates a movement plot where specific colors represent specific limb orientations. Linguists can use these plots to visually identify rhythm, duration, and repetition across a timeline, bridging the gap between machine data and human observation.

Experimental Evaluation

The method was evaluated on two datasets: the JSL Corpus (Japanese Sign Language), used to test sociolinguistic features and robustness against perspective shifts, and the WLASL-100 (Word-Level American Sign Language), a public benchmark for isolated sign recognition.

Key Results

  • State-of-the-Art Performance: On the WLASL-100 benchmark, the 3D normalization achieved a Top-1 accuracy of 84.26%, outperforming previous pose-based models.
  • Viewpoint Robustness: In the JSL dataset, the 3D method maintained significant accuracy when training on front-view and testing on side-view data (32.73% for Lexical Signs), whereas standard 2D normalization performance dropped to 8.18%.
  • Sociolinguistic Insights: The model successfully identified regional "accents" by clustering signers based on their prefecture and school.

Discussion and Insights

A significant finding is the existence of "Movement Fingerprints." The system identified individual signers with over 72% accuracy based solely on movement patterns, suggesting that signing style is a unique biometric identifier. This has implications for the privacy of "anonymized" sign language corpora.

Limitations

The system currently relies on standard human body proportion assumptions, which may not apply to all individuals. Additionally, accuracy can decrease during joint occlusion (e.g., hands touching the face). Future work aims to better integrate facial nuances, such as mouthings and eyebrow movements, which are critical for sign language grammar.

Conclusion

By transforming 2D video into normalized 3D skeletons and then into intuitive RGB plots, this research makes sign language data more accessible for both machines and linguists. It provides a robust framework for cross-dataset recognition and opens new avenues for sociolinguistic research.

Summary updated

BibTeX
@inproceedings{skobov2023making,
  author = {Victor Skobov and Mayumi Bono},
  title = {{Making Body Movement in Sign Language Corpus Accessible for Linguists and Machines with Three-Dimensional Normalization of MediaPipe}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.124},
}
Figure