DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation

Biyi Fang, Jillian Co, Mi ZhangSenSys 2017 · DOI

TL;DR

This paper introduces DeepASL, a novel system for real-time American Sign Language (ASL) translation that is both non-intrusive and capable of translating full sentences. By leveraging infrared sensing and a hierarchical deep learning architecture, DeepASL significantly breaks down communication barriers for the deaf community.

What Problem Does This Paper Address?

The communication gap between deaf individuals who use sign language and hearing individuals who do not is a significant societal challenge. While sign language translation technology aims to bridge this gap, existing systems often suffer from several limitations: they can be intrusive (requiring sensors attached to the body), constrained by lighting conditions, or only capable of translating single signs at a time, which hinders natural conversation flow. This paper addresses the need for a practical, non-intrusive, and sentence-level ASL translation system that can be seamlessly integrated into daily life.

Methods

DeepASL employs a multi-stage approach:

  • Non-Intrusive Sensing: The system utilizes the Leap Motion controller, which uses infrared light to capture hand and finger movements without requiring the user to wear any sensors. This allows for a natural and unobtrusive signing experience.
  • ASL Characteristics Extraction: The raw skeleton joint data from Leap Motion is processed to extract key ASL characteristics: hand shape, hand movement, and the relative location of the hands.
  • Hierarchical Bidirectional RNN (HB-RNN): A novel HB-RNN architecture is introduced to model the complex spatio-temporal dynamics of the extracted ASL characteristics. This hierarchical structure allows for the integration of low-level features into a high-level representation suitable for word-level translation. The bidirectional nature of the RNN helps capture context from both past and future movements, crucial for disambiguating similar signs.
  • Connectionist Temporal Classification (CTC): For sentence-level translation, DeepASL adopts a CTC-based framework. This approach eliminates the need for pre-segmenting sentences into individual words, enabling end-to-end translation and allowing users to sign naturally without pauses between words.

Datasets

The authors collected and utilized three datasets for their research:

  • ASL Words Dataset: This dataset comprises 56 commonly used ASL words (29 two-handed, 27 one-handed), with approximately 10 samples per word from 11 participants.
  • ASL Sentences Dataset: This dataset includes 100 meaningful ASL sentences, generated based on a sentence template, with data collected from 11 participants.
  • In-the-Field Dataset: This dataset was collected under various real-world conditions, including different lighting (poor indoor, bright outdoor), body postures (walking, standing), and interference sources (people, multiple devices), to test the system's robustness.

These datasets are noted as being among the largest and most comprehensive in the sign language translation literature.

Key Insights and Ideas

  • Infrared Sensing for ASL: The paper highlights the effectiveness of infrared light-based sensing (via Leap Motion) for capturing ASL signs, demonstrating its robustness to lighting conditions and its non-intrusive nature.
  • Hierarchical Bidirectional RNN for ASL: The HB-RNN architecture proves crucial for effectively modeling the intricate spatial and temporal aspects of ASL signs, outperforming simpler models.
  • CTC for End-to-End Sentence Translation: The adoption of CTC is a significant step towards practical ASL translation, as it removes the cumbersome requirement of word pre-segmentation and allows for natural, continuous signing.
  • Robustness in Real-World Scenarios: DeepASL demonstrates impressive accuracy even under challenging conditions like poor lighting and the presence of other people, suggesting its potential for real-world deployment.

Key Results

  • Word-Level Accuracy: DeepASL achieves an average word-level translation accuracy of 94.5% across 56 ASL words, with a low standard deviation, indicating good generalization across users.
  • Sentence-Level Performance: For unseen ASL sentences, the system achieves an average word error rate (WER) of 8.2% (on unseen participants) and 16.1% (on unseen users), which is considered very promising.
  • Robustness: The system maintains high accuracy (over 91.8%) across various lighting conditions, body postures, and interference scenarios.
  • Efficiency: DeepASL demonstrates real-time performance, with sentence recognition taking less than 282 ms on a mobile CPU, making it suitable for interactive applications.

Limitations and Future Work

  • Dataset Size: While the datasets are substantial, they still cover only a fraction of the ASL vocabulary. The authors acknowledge the need for more comprehensive data and have initiated a crowdsourcing effort to address this.
  • AR Headset Burden: The current prototype for the "Personal Interpreter" application uses a tablet and AR headset, which can be burdensome. Future work aims to miniaturize AR headsets for a more seamless user experience.
  • Language Model Integration: The current sentence-level translation does not explicitly use a language model, which could potentially further improve accuracy.
  • Generalizability to Other Sign Languages: While the approach is presented for ASL, its generic nature in modeling hand gestures suggests potential for adaptation to other sign languages.

Summary updated

BibTeX
@inproceedings{fang2017deepasl,
  author = {Biyi Fang and Jillian Co and Mi Zhang},
  title = {{DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation}},
  booktitle = {SenSys},
  year = {2017},
  doi = {10.1145/3131672.3131693},
}
Figure