Advanced speech communication system for deaf people

Rubén San-Segundo, Verónica López, Raquel Martín, Syaheerah Lufti, Javier Ferreiros, Ricardo Córdoba, José Manuel PardoInterspeech 2010

TL;DR

This paper presents an advanced speech communication system designed to bridge the communication gap between deaf individuals and hearing people in real-world government service interactions, specifically focusing on driver's license renewal. The system translates spoken Spanish to Spanish Sign Language (LSE) and vice-versa, integrating multiple translation strategies and a 3D avatar for sign representation, and was successfully evaluated in a field trial.

What Problem Does This Paper Address?

Deaf individuals in Spain face significant communication barriers when interacting with government services, largely due to a lack of widespread knowledge of Spanish Sign Language (LSE) among government employees. This disparity hinders their access to essential services and opportunities. The paper addresses this by developing and evaluating a technological solution that facilitates two-way communication: translating spoken Spanish into LSE for deaf users and translating LSE (specified by users) into spoken Spanish for government employees.

Methods

The system comprises two main modules:

  • Spanish to LSE Translation:
    • Speech Recognition: A speaker-independent HMM-based continuous speech recognition system with confidence measures and adaptation capabilities.
  • Natural Language Translation: A hierarchical module combining three strategies:
  • Example-Based: Translates based on similarity to existing translated examples.
  • Rule-Based: Uses expert-defined translation rules.
  • Statistical: Employs phrase-based and Finite State Transducer approaches. The system dynamically chooses between example-based and a background module (rule-based or statistical) based on a calculated ratio of glosses to words.

  • Sign Animation: A 3D avatar (VGuido) animates the generated LSE sign sequence.

  • LSE to Spanish Generation:
    • Visual Interface: Allows deaf users to specify LSE sign sequences using glosses, with tools for sign specification, date/time definition, and spelling. Signs can be searched using specific characteristics.
  • Language Translation: A hierarchical module similar to the Spanish-to-LSE module, combining example-based, rule-based, and statistical translation strategies to convert sign sequences into word sequences.
  • Text-to-Speech: A commercial Text-to-Speech converter (Loquendo) generates spoken Spanish from the translated word sequence.

Datasets

The primary dataset was collected in collaboration with the Local Traffic Office in Toledo, Spain, over three weeks. It includes frequent explanations from government employees and questions from users.

  • Initial Corpus: 707 sentences (547 from employees, 160 from users).
  • Expanded Corpus: 2,124 sentences by incorporating variants of Spanish sentences and their LSE translations.
  • Sign Representation: A database of over 400 signs was generated, including descriptions in glosses, SEA, HamNoSys, and SIGML.

Key Insights and Ideas

  • Hybrid Translation Strategies are Crucial: Combining example-based, rule-based, and statistical methods offers a robust approach to sign language translation, leveraging the strengths of each. The dynamic selection based on input characteristics is a key innovation.
  • Real-World Domain Evaluation is Essential: Testing the system in a practical setting like a driver's license renewal office provides invaluable feedback on usability and performance beyond laboratory conditions.
  • Avatar Naturalness is a Major Hurdle: While technical translation accuracy can be high, the naturalness and expressiveness of the 3D avatar for sign language representation remain a significant challenge for user acceptance.
  • Standardization of LSE is Paramount: The lack of a fully standardized gloss notation for LSE complicates translation and user input, highlighting the need for continued efforts in this area.

Key Results

Spanish into LSE Translation (Field Evaluation): * Speech Recognition Word Error Rate (WER): 4.6% (improved from lab tests due to adaptation). * Sign Error Rate (SER) after translation: 8.5% (also improved). * Average Translation Time: 4.6 seconds, enabling fluent dialogue. * Example-based translation used in 95.0% of cases, indicating the reliability of the linguistic study.

LSE into Spanish Generation (Field Evaluation): * Translation Error Rate: 2.56% (good performance). * Average Translation Time: 0.001 sec. * Average Text-to-Speech Conversion Time: 1.7 sec. * Example-based strategy used in 92.7% of cases. * User needed less than 20 seconds to specify a gloss sequence, considered reasonable given limited practice time.

Subjective User Feedback: * Government employee: Positive score (3.5) for the speech-LSE system, but found the single screen uncomfortable. * Deaf users: Lower scores for the speech-LSE system (overall 2.6), particularly for sign naturalness (1.6). For the LSE-speech system, users gave reasonable scores but noted interface complexity and gloss standardization issues.

Limitations and Future Work

Limitations: * Avatar Naturalness: The naturalness of the 3D avatar for sign language representation is a significant area for improvement. * LSE Standardization: The lack of complete standardization in LSE gloss notation poses challenges for both translation and user input. * User Interface Complexity: Some users found the LSE-speech system's interface complex. * Single Screen Issue: The government employee found the single screen setup for the speech-LSE system uncomfortable.

Future Work: * Continue improving avatar naturalness and expressiveness. * Increase efforts in standardizing LSE. * Incorporate a sign selection mechanism based on images or GIFs to aid users. * Consider using two screens for the speech-LSE system to improve user comfort. * Further refine the translation strategies and corpus.

Summary updated

BibTeX
@inproceedings{sansegundo2010advanced,
  author = {Rubén San-Segundo and Verónica López and Raquel Martín and Syaheerah Lufti and Javier Ferreiros and Ricardo Córdoba and José Manuel Pardo},
  title = {{Advanced speech communication system for deaf people}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure