Sign language recognition based on dual-path background erasure convolutional neural network

Junming Zhang, Xiaolong Bu, Yushuai Wang, Hao Dong, Yu Zhang, Haitao WuScientific 2024 · DOI

Abstract

Sign language recognition is a critical field for enhancing accessibility, yet many current systems are hindered by high computational costs or reliance on expensive sensors. This paper introduces the Dual-path Background Erasure Convolutional Neural Network (DPCNN), a lightweight model designed to isolate hand gestures by explicitly modeling and subtracting background noise. Evaluated on the ASL Finger Spelling dataset, the DPCNN achieves a 99.52% accuracy and a Macro-F1 score of 0.997. With only 60,280 parameters, the model is optimized for deployment on resource-constrained mobile and IoT devices.

Introduction

Effective sign language recognition (SLR) systems often face a trade-off between accuracy and accessibility. High-performance systems frequently utilize depth sensors or wearable gloves, which are costly and limit portability. Conversely, vision-based systems using standard RGB cameras often struggle with "noisy" real-world backgrounds or require massive deep learning architectures that exceed the processing power of mobile devices. This study proposes a novel DPCNN architecture that addresses these challenges through a physically motivated "background erasure" mechanism.

Methodology: Dual-Path Background Erasure

The core innovation of the DPCNN is its dual-path architecture, which processes input images through two parallel pipelines:

  1. The Primary Path: This path extracts "overall features," encompassing both the hand gesture and the surrounding environment. It utilizes Average Pooling to capture global data trends.
  2. The Auxiliary Path: Specifically designed to model the background, this path uses Max Pooling to identify the most salient local features, such as the sharp edges of background objects.

The Erasure Mechanism

At four distinct stages within the network, the features extracted by the auxiliary path are subtracted from those in the primary path ($M_{main} - M_{auxiliary}$). This explicit subtraction "erases" the background noise, leaving a refined representation of the hand gesture for final classification. The resulting features are flattened and passed through a fully connected layer (128 units) before reaching the final 24-unit output layer.

Experimental Setup and Datasets

The model was benchmarked using the ASL Finger Spelling dataset, which consists of approximately 65,702 RGB images across 24 static alphabetic categories (A-Y, excluding dynamic gestures 'J' and 'Z').

Example of the 'W' gesture from the ASL Finger Spelling dataset, showing a complex indoor background.

Unlike laboratory datasets with uniform backgrounds, this dataset features complex, non-uniform environments, providing a rigorous test for the model's generalization capabilities.

Example of the 'A' gesture, illustrating the non-uniform environments the DPCNN must process.

Results and Discussion

The DPCNN demonstrated state-of-the-art performance while maintaining an exceptionally small footprint:

  • Accuracy and Precision: The model achieved a total accuracy of 99.52% and a Macro-F1 score of 0.997, indicating high reliability across all gesture categories.
  • Efficiency: The architecture contains only 60,280 parameters (0.06M). This is significantly smaller than alternative models, which range from 1.88M to 21.24M parameters, making the DPCNN ideal for "small terminals" like smartphones.
  • Ablation Study: Comparisons between feature subtraction and standard feature addition confirmed that "erasing" noise is superior to merging it with the signal, resulting in lower loss and higher accuracy.

Conclusion

The DPCNN provides a robust, lightweight solution for vision-based sign language recognition. By leveraging a dual-path architecture to subtract background noise, the model achieves high accuracy on complex datasets without the need for massive parameter counts. Future work will focus on extending this "erasure" concept to dynamic gestures and full-sentence recognition.

Summary updated

Abstract

Sign language is an important way to provide expression information to people with hearing and speaking disabilities. Therefore, sign language recognition has always been a very important research topic. However, many sign language recognition systems currently require complex deep models and rely on expensive sensors, which limits the application scenarios of sign language recognition. To address this issue, based on computer vision, this study proposed a lightweight, dual-path background erasing deep convolutional neural network (DPCNN) model for sign language recognition. The DPCNN consists of two paths. One path is used to learn the overall features, while the other path learns the background features. The background features are gradually subtracted from the overall features to obtain an effective representation of hand features. Then, these features are flatten into a one-dimensional layer, and pass through a fully connected layer with an output unit of 128. Finally, use a fully connected layer with an output unit of 24 as the output layer. Based on the ASL Finger Spelling dataset, the total accuracy and Macro-F1 scores of the proposed method is 99.52% and 0.997, respectively. More importantly, the proposed method can be applied to small terminals, thereby improving the application scenarios of sign language recognition. Through experimental comparison, the dual path background erasure network model proposed in this paper has better generalization ability.
BibTeX
@inproceedings{zhang2024sign,
  author = {Junming Zhang and Xiaolong Bu and Yushuai Wang and Hao Dong and Yu Zhang and Haitao Wu},
  title = {{Sign language recognition based on dual-path background erasure convolutional neural network}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-62008-z},
}
Figure