WW-CSL: A New Dataset for Word-Based Wearable Chinese Sign Language Detection

Fan Xu, Kai Liu, Yifeng Yang, Keyu YanCOLING 2024

TL;DR

This paper introduces WW-CSL, a novel dataset for Chinese Sign Language (CSL) detection, featuring a unique three-form representation (sensor data, video, and text) and a new wearable glove. They also propose MM-CSLD, a multimodal transformer-based framework that effectively fuses wearable sensor data with video-based skeleton information for improved CSL detection.

What Problem Does This Paper Address?

Sign language is a vital communication method for the hearing-impaired. However, developing accurate sign language detection systems faces challenges. Traditional video-based methods struggle with poor lighting and occlusions. While wearable glove-based systems offer robustness, existing datasets are often small and limited in scope. Crucially, there's a lack of research that effectively combines the strengths of both video and wearable sensor data for sign language recognition, particularly for Chinese Sign Language (CSL), which has fewer publicly available resources compared to languages like American Sign Language (ASL). This paper aims to bridge this gap by creating a comprehensive CSL dataset and a multimodal detection framework.

Methods

The core of this work lies in two main contributions: a new dataset and a novel detection framework.

WW-CSL Dataset: The dataset is built around a custom wearable glove equipped with 2 Inertial Measurement Units (IMUs) and 10 flex sensors. This glove captures sequential sensor data from finger movements. To create a richer representation, the authors also incorporate:

  • Gesture Video: Existing video data from a large-scale CSL corpus is used.
  • Gesture Text: The corresponding word or phrase for each gesture.

This "three-form" representation allows for a more comprehensive understanding of CSL gestures. The dataset comprises 3,000 samples covering 100 distinct word-based CSL gestures.

MM-CSLD Framework: The proposed Multi-Modal CSL Detection (MM-CSLD) framework leverages a transformer architecture. It fuses two types of input:

  • Wearable Sensor Data: Local, sequential data from the IMUs and flex sensors.
  • Video-Based Skeleton Data: Global, fine-grained skeleton representations extracted from gesture videos.

The framework uses a transformer encoder-decoder structure. The input data is first preprocessed (sparse sampling, segmentation) and then fused. Positional encoding is applied to both sensor and skeleton data before being fed into the transformer. The final output is a predicted Chinese sign language label.

Datasets

  • WW-CSL (New Dataset): Introduced in this paper. It contains 3,000 samples of 100 word-based CSL gestures, represented in three forms: sequential sensor data (from a custom wearable glove), gesture video, and gesture text.
  • Huang et al. (2018) CSL Dataset: Used for gesture video data. The authors selected 100 common words from this existing large-scale video-based CSL corpus.

Key Insights and Ideas

  • Multimodality is Key: Combining wearable sensor data with video-based skeleton information significantly improves CSL detection performance. This suggests that different modalities capture complementary aspects of sign language.
  • Transformer Architecture's Suitability: The transformer's ability to model long-range dependencies and relationships makes it well-suited for both sequential sensor data and the temporal nature of sign language gestures.
  • Wearable Glove Design: The integration of IMUs and flex sensors on a glove provides a robust way to capture fine-grained finger movements, overcoming some limitations of purely vision-based approaches.
  • Data Representation Matters: The "three-form" data representation (sensor, video, text) offers a more complete picture of a gesture, potentially leading to better model training and understanding.
  • Number of Transformer Heads: The study found that using too many heads (17) in the transformer's self-attention mechanism can be detrimental, possibly due to the limited input dimensionality of the skeleton data. A smaller number (2) performed better in this specific scenario.

Key Results

The MM-CSLD framework achieved state-of-the-art results on the newly introduced WW-CSL dataset.

  • Combined Performance: The MM-CSLD model achieved an accuracy of 97.56% when combining both wearable sensor data and video-based skeleton data.
  • Wearable-Only Performance: When using only the wearable sensor data, the MM-CSLD model (without video data) achieved an accuracy of 93.99%, outperforming other baseline models like GRU and LSTM.
  • Video-Only Performance: When using only video-based skeleton data, the MM-CSLD model (without sensor data) achieved an accuracy of 72.09%, which is still competitive but highlights the benefit of sensor fusion.
  • Baseline Comparisons: The transformer-based MM-CSLD consistently outperformed traditional sequential models (GRU, LSTM) and non-sequential models (CNN, Dense) across different data modalities.

Limitations and Future Work

Limitations: * The experiments were conducted on a specific dataset (Chinese sign language) and may not generalize perfectly to other sign languages or scenarios. * The current dataset is word-based, and the generalizability to different datasets and languages needs further exploration.

Future Work: * The authors plan to focus on sentence-based CSL data collection. * They aim to develop continuous CSL detection systems.

Summary updated

BibTeX
@inproceedings{xu2024ww,
  author = {Fan Xu and Kai Liu and Yifeng Yang and Keyu Yan},
  title = {{WW-CSL: A New Dataset for Word-Based Wearable Chinese Sign Language Detection}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure