Global-local Enhancement Network for NMFs-aware Sign Language Recognition

Hezhen Hu, Wengang Zhou, Junfu Pu, Houqiang Li

Abstract

Sign language recognition (SLR) is a challenging problem, involving complex manual features, i.e., hand gestures, and fine-grained non-manual features (NMFs), i.e., facial expression, mouth shapes, etc. Although manual features are dominant, non-manual features also play an important role in the expression of a sign word. Specifically, many sign words convey different meanings due to non-manual features, even though they share the same hand gestures. This ambiguity introduces great challenges in the recognition of sign words. To tackle the above issue, we propose a simple yet effective architecture called Global-local Enhancement Network (GLE-Net), including two mutually promoted streams towards different crucial aspects of SLR. Of the two streams, one captures the global contextual relationship, while the other stream captures the discriminative fine-grained cues. Moreover, due to the lack of datasets explicitly focusing on this kind of features, we introduce the first non-manual-features-aware isolated Chinese sign language dataset (NMFs-CSL) with a total vocabulary size of 1,067 sign words in daily life. Extensive experiments on NMFs-CSL and SLR500 datasets demonstrate the effectiveness of our method.

Introduction

Most Sign Language Recognition (SLR) systems treat the task primarily as a hand-tracking or gesture-recognition problem. However, sign languages are highly expressive and rely heavily on Non-Manual Features (NMFs)—such as facial expressions, mouth shapes, eye gaze, and head tilts. In Chinese Sign Language (CSL), for example, the signs for "reciprocate" and "revenge" use the exact same hand motions but differ entirely in facial expression.

Global view of a signer where non-manual features (mouth shape) are highlighted for recognition.

Previous datasets and models largely ignore these fine-grained, highly confusing pairs. Because standard 3D Convolutional Neural Networks (CNNs) tend to lose small spatial details during pooling operations, they struggle to differentiate between words where the only difference is a subtle mouth movement.

Methodology: The GLE-Net Architecture

The authors propose the Global-local Enhancement Network (GLE-Net), a two-stream architecture built on a 3D-CNN backbone (like ResNet-50).

Global Enhancement Module

This stream captures long-range spatio-temporal context. It uses a self-attention mechanism to aggregate features across all positions, helping the network understand the holistic movement of the signer while filtering out irrelevant background noise.

Local Enhancement Module

This stream focuses on the tiny, fine-grained details. Instead of standard pooling, it uses a saliency map generated from the global stream to perform adaptive sampling. This essentially "zooms in" on discriminative regions, such as the mouth or eyes, ensuring they are not lost in deeper network layers.

Local enhancement focusing on fine-grained mouth features to resolve ambiguity in sign recognition.

Mutual Promotion Module

Instead of training streams independently, a mutual learning strategy is used. By minimizing the Kullback-Leibler (KL) divergence between the probability distributions of the two streams, the global and local streams "teach" each other during training, resulting in a more robust final model.

Datasets and Results

The research introduces NMFs-CSL, the first isolated Chinese Sign Language dataset explicitly focused on non-manual features. It contains 32,010 RGB videos covering 1,067 sign words, including 610 "confusing words" that share hand gestures but differ in NMFs.

GLE-Net achieved 69.9% Top-1 accuracy on NMFs-CSL, outperforming state-of-the-art models like Slowfast (66.3%) and I3D (64.4%). The performance gap was most significant on the "confusing" subset, where GLE-Net beat Slowfast by a 4.8% margin (51.8% vs 47.0%). On the existing SLR500 dataset, the model achieved 96.8% Top-1 accuracy using only RGB data, surpassing previous multi-modal methods (RGB + Depth + Skeleton) which peaked at 88.7%.

Conclusion

The GLE-Net architecture demonstrates that paying attention to the face is just as important as watching the hands in sign language recognition. By combining global context with local fine-grained details, the model successfully resolves ambiguities in "confusing" sign pairs. Future work includes extending these NMF-aware techniques to continuous sign language translation and exploring more sophisticated stream fusion strategies.

Summary updated

Abstract

Sign language recognition (SLR) is a challenging problem, involving complex manual features, i.e., hand gestures, and fine-grained non-manual features (NMFs), i.e., facial expression, mouth shapes, etc. Although manual features are dominant, non-manual features also play an important role in the expression of a sign word. Specifically, many sign words convey different meanings due to non-manual features, even though they share the same hand gestures. This ambiguity introduces great challenges in the recognition of sign words. To tackle the above issue, we propose a simple yet effective architecture called Global-local Enhancement Network (GLE-Net), including two mutually promoted streams towards different crucial aspects of SLR. Of the two streams, one captures the global contextual relationship, while the other stream captures the discriminative fine-grained cues. Moreover, due to the lack of datasets explicitly focusing on this kind of features, we introduce the first non-manual-features-aware isolated Chinese sign language dataset~(NMFs-CSL) with a total vocabulary size of 1,067 sign words in daily life. Extensive experiments on NMFs-CSL and SLR500 datasets demonstrate the effectiveness of our method.
BibTeX
@article{hu2021global,
  author = {Hezhen Hu and Wengang Zhou and Junfu Pu and Houqiang Li},
  title = {{Global-local Enhancement Network for NMFs-aware Sign Language Recognition}},
  journal = {ACM Trans. Multim. Comput. Commun. Appl.},
  year = {2021},
  url = {https://openreview.net/forum?id=VGH9YjNi61},
}
Figure