AI-Driven Sign Language Interpretation for Nigerian Children at Home

Ifeoma Nwogu, Roshan L. Peiris, Karthik Dantu, Ruchi Gamta, Emma AsonyeIJCAI 2023 · DOI

TL;DR

This paper proposes an AI-driven, two-way sign language interpretation tool to bridge the communication gap between Deaf and Hard of Hearing (DHH) children and their hearing families in Nigeria. By leveraging neural deep learning, the system aims to improve social inclusion and educational opportunities for DHH children, addressing a critical need in a region where such support is often lacking.

What Problem Does This Paper Address?

In Nigeria, a significant number of school-aged children suffer from severe to profound hearing loss. Many of these children acquire their hearing loss later in life, meaning their parents are typically hearing. While these children may communicate effectively with teachers using American Sign Language (ASL) in school, communication at home with family members is often challenging or nonexistent. This lack of communication leads to adverse social consequences, including stigmatization, exclusion, and limited future opportunities for DHH individuals. The paper highlights that a substantial portion of the deaf population in Nigeria remains under-educated and economically underdeveloped, underscoring the link between deafness and poverty. The project aims to address this by creating an AI tool that facilitates natural language communication within the home environment.

Methods

The project proposes a multi-faceted approach centered around an AI-driven, two-way sign language interpretation tool. Key methodological components include:

  • User-Centered Interface Design: An iterative co-design process involving DHH students, hearing students, instructors, and interpreters to develop a functional and user-friendly interface.
  • Two-Way Neural ASL Translation: This is the core technical contribution. The system will involve:
  • Sign-to-Gloss Translation: Developing robust neural models to translate sign language into a textual representation (gloss).
  • Gloss-to-Sign Translation: A more challenging task, focusing on generating sign language from gloss.
  • Auxiliary Gloss-to-English and English-to-Gloss Models: To facilitate broader translation capabilities.
  • Dual Learning Paradigm: Employing a dual learning mechanism to enhance both sign-to-text and text-to-sign translation by jointly training the models.
  • Avatar Development: Creating a signing avatar that displays facial expressions, with the goal of eventually coordinating these with manual signs for more naturalistic communication.
  • Timely Realization of the Interpretation Device: Addressing technical challenges related to detecting, tracking, and following communicators for real-time interpretation, and ensuring the system can operate efficiently on small, mobile devices suitable for low-resource environments.

Datasets

The project plans to utilize and create several datasets:

  • Multicultural Word-Level ASL Dataset: A new dataset will be created for Nigeria, using the sign language version of the Swadesh list.
  • Existing Word-Level ASL Dataset: The project will leverage a 2000-word ASL dataset (from Li et al., 2020) containing over 21,000 video samples of isolated signed words.
  • How2Sign Dataset: A large, continuous ASL dataset (35,000 video samples) annotated with glosses, used for training.
  • ASLing Dataset: A diverse, real-life ASL dataset collected at RIT, consisting of over 1,200 video samples of signed phrases in uncontrolled, everyday situations.
  • Regional Data: Approximately 200 signed phrases and 200 word videos will be collected from local Nigerian signers for implantation testing.

Key Insights and Ideas

  • Bridging the Home Communication Gap: The project directly addresses a critical social and educational gap by focusing on home-based communication, which is often overlooked in favor of school-based interventions.
  • Dual Learning for Enhanced Translation: The application of a dual learning paradigm for both sign-to-text and text-to-sign translation is a novel approach to improve the robustness and accuracy of the system.
  • Leveraging Word Embeddings for Unseen Signs: The idea of using word embeddings to extrapolate signs for words not explicitly present in the training data is a promising strategy for handling vocabulary expansion.
  • Focus on Gloss-to-Sign Production: While sign-to-gloss is gaining traction, the paper highlights the greater challenge and importance of sign production (gloss-to-sign), which is crucial for generating visual output.
  • Inclusivity and SDG Alignment: The project is explicitly linked to UN Sustainable Development Goals (SDGs) such as Quality Education, Decent Work and Economic Growth, and Reduced Inequalities, framing it as a project with significant societal impact.

Key Results

The paper is a proposal and outlines the planned research. Therefore, concrete quantitative results are not yet available. However, the authors state the following:

  • Sign-to-Text (S2T) Performance: Their current S2T application achieved a BLEU1 score of 22.39 and BLEU4 of 12.25 on their ASLing dataset. They note these scores are lower than those achieved on larger, more established datasets like RWTH PHOENIX-Weather-2014T, attributing this to the limited size of their current annotated continuous ASL datasets.
  • Text-to-Sign (T2S) Capability: They can currently produce up to 2000 words from an ASL word-level dataset. The current iteration is limited to single words due to technical constraints.
  • Usability Goals: The project aims for usability metrics including effectiveness (accuracy and completeness), efficiency (resource usage), and satisfaction (comfort and acceptability).
  • Target Metrics: For the implantation test, they aim for BLEU1-4 scores in a similar range to their US-based ASL translation scores and an accuracy of greater than 75% on the 200 regional words.

Limitations and Future Work

  • Avatar Limitations: The current interpretation avatar is limited to manual signs and does not fully incorporate non-manual features (facial expressions, gaze, torso movements), which are crucial for natural sign language. While facial expressions are being added, their coordination with manual signs is a future goal.
  • Regional Dialects and Indigenous Sign Languages: ASL is the primary language taught, but regional "dialects" and indigenous sign languages exist in Nigeria. The system, initially trained on American ASL, may face challenges with these variations, though iterative updates with Nigerian data are planned.
  • Speech-to-Text/Text-to-Speech Compatibility: Commercial speech-to-text and text-to-speech converters may struggle with Nigerian English accents and intonations. The project plans to explore converters that can train on users' voices.
  • Non-Native English Speakers: Many target families may not be native English speakers, and Hausa is prevalent in Kaduna. The initial cohort will consist of families where parents speak English, but future work may involve integrating English-Hausa/Hausa-English translation.
  • Future Work: The project will continue to improve the tool based on usability testing and field evaluation. Future work will focus on incorporating non-manual features into the avatar, addressing regional language variations, and expanding the system's capabilities to handle a wider range of languages and accents.

Summary updated

BibTeX
@inproceedings{nwogu2023ai,
  author = {Ifeoma Nwogu and Roshan L. Peiris and Karthik Dantu and Ruchi Gamta and Emma Asonye},
  title = {{AI-Driven Sign Language Interpretation for Nigerian Children at Home}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/710},
}
Figure