Mutual Support of Data Modalities in the Task of Sign Language Recognition

Ivan Gruber, Zdenek Krnoul, Marek Hrúz, Jakub Kanis, Matyas BohacekWorkshops @ CVPR 2021 · OpenReview

Abstract

This paper presents a method for automatic sign language recognition that was utilized in the CVPR 2021 ChaLearn Challenge (RGB track). Our method is composed of several approaches combined in an ensemble scheme to perform isolated sign-gesture recognition. We combine modalities of video sample frames processed by a 3D ConvNet (I3D), with body-pose information in the form of joint locations processed by a Transformer, hand region images transformed into a semantic space, and linguistically defined locations of hands. Although the individual models perform sub-par (60% to 93% accuracy on validation data), the weighted ensemble results in 95.46% accuracy.

Introduction

Sign Language (SL) is a complex, multi-modal communication form where meaning is conveyed through a mix of hand shapes, motion, body posture, and facial expressions. Recognizing these signs "in the wild" is notoriously difficult because representative cues must be extracted directly from raw pixels, which are sensitive to lighting, background clutter, and camera angles.

While standard action recognition models perform decently, they often miss the fine-grained linguistic nuances of SL. The authors address this by building a system where different models, each specializing in a specific data modality (pixels, joints, or semantic hand poses), support each other to overcome individual weaknesses.

Methods

The authors' pipeline is an ensemble of three primary model types, optimized via the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to find the optimal weighting for each modality.

1. I3D (3D ConvNets)

These models process raw RGB video frames. The authors utilized 13 different ResNet50-I3D models. To improve focus, the system uses pose information to crop the video around the signer, sometimes applying masking to isolate relevant regions. Some models are trained on random frames, while others target "key-frames" where motion is minimal, representing the semantic peak of a sign.

Overview of the I3D model pipeline utilizing pose-guided video cropping.

2. Pose-Transformer

Using OpenPose, the system extracts 2D coordinates for body joints. These coordinates are fed into a Transformer model (inspired by the Vision Transformer) to classify the sign based purely on skeletal motion. Despite I3D models having higher individual accuracy, this Pose-transformer received the highest weight in the final ensemble, suggesting skeletal motion provides a stable global signal.

3. VLE-Transformer (Visual Language Embedding)

The authors introduced a novel Visual Language Embedding (VLE). They clustered thousands of hand images into 39 distinct "hand poses" and trained a MobileNet to map hand crops into a semantic space. This ensures that similar hand shapes are represented by similar vectors, making the model robust to variations between signers. This VLE data is combined with Location Vectors—15 linguistically defined regions (e.g., "above the head," "near the nose")—and processed by a second Transformer.

The VLE Transformer branch, fusing semantic hand embeddings with linguistic location vectors.

Key Results and Insights

The system was evaluated on the AUTSL dataset, containing over 32,000 videos of 226 signs.

  • Synergy over Individual Performance: Individual models achieved accuracies between 60% and 93%. However, the optimized ensemble reached 95.46% accuracy on the test set. This demonstrates that the models make complementary errors.
  • Ensemble Optimization: The CMA-ES optimized ensemble significantly outperformed a simple equal-weight ensemble (94.66%), highlighting the importance of modality-specific trust levels.
  • Semantic Mapping and Linguistic Priors: Incorporating VLE and Location modalities boosted performance by nearly 0.4%. Mapping images to a semantic space and using domain-specific body regions helps distinguish signs that are visually similar but occur in different signing spaces.

Conclusion and Future Work

While the system achieves high accuracy, it is currently constrained by 2D joint detections, which struggle with depth-based motion. Future work includes integrating 3D pose estimation and optical flow to further capture motion dynamics. Additionally, addressing the class imbalance in hand pose clusters could further refine the semantic embeddings.

Summary updated

Abstract

This paper presents a method for automatic sign language recognition that was utilized in the CVPR 2021 ChaLearn Challenge (RGB track). Our method is composed of several approaches combined in an ensemble scheme to perform isolated sign-gesture recognition. We combine modalities of video sample frames processed by a 3D ConvNet (I3D), with body-pose information in the form of joint locations processed by a Transformer, hand region images transformed into a semantic space, and linguistically defined locations of hands. Although the individual models perform sub-par (60% to 93% accuracy on validation data), the weighted ensemble results in 95.46% accuracy.
BibTeX
@inproceedings{gruber2021mutual,
  author = {Ivan Gruber and Zdenek Krnoul and Marek Hrúz and Jakub Kanis and Matyas Bohacek},
  title = {{Mutual Support of Data Modalities in the Task of Sign Language Recognition}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=m8HIA4nOhm},
}
Figure