TL;DR
This paper introduces a user-centered methodology for adapting speech-to-sign language translation systems to new domains, specifically a hotel reception. By involving end-users in the design and leveraging a parallel corpus, they achieved high translation accuracy and speed, demonstrating a practical approach to bridging communication gaps for the deaf community.
What Problem Does This Paper Address?
Communication barriers between deaf and hearing individuals remain a significant challenge. While automatic translation systems for speech-to-sign language exist, adapting them to specific real-world domains, like a hotel reception, requires a structured approach. Existing systems often struggle with domain-specific vocabulary and nuances, leading to lower accuracy and usability. This paper tackles the problem of efficiently and effectively adapting such a system to a new, practical domain, ensuring it meets the needs of both deaf customers and hearing staff.
Methods
The core of this paper is a novel, user-centered methodology for domain adaptation. It's built upon the Participatory Design approach and comprises four key phases:
- Requirement Analysis: This involves understanding the specific needs of deaf customers and hotel receptionists through workshops. Both user and technical requirements are defined.
- Parallel Corpus Generation: A crucial step where a domain-specific parallel corpus (Spanish speech and its corresponding LSE sign language translation) is created. This corpus is essential for training and adapting the system.
- Technology Adaptation: This phase focuses on adapting the existing speech-to-sign language translation technology. This includes:
- Speech Recognition: Adapting acoustic and language models using the new domain's vocabulary.
- Language Translation: Employing an example-based translation strategy, augmented by a statistical background module, to translate Spanish sentences into LSE sign sequences. Generalization techniques are used to improve the effectiveness of translation examples.
- Sign Language Representation: Utilizing an XML-based declarative abstraction for sign animation, with an improved sign editor for faster specification.
- Speech Generation from LSE: Integrating the LSESpeak system, which translates LSE into spoken Spanish, and also includes an SMS-to-Spanish translation utility.
- Field Evaluation: The final step involves testing the adapted system with real users in a realistic scenario to assess its performance and usability.
Datasets
The paper introduces a new domain-specific parallel corpus for the hotel reception domain. This corpus was collected through collaboration with the Hotel "Intur Palacio de San Martín" in Madrid. It contains over 1,000 sentences, manually analyzed and selected, totaling 500 core sentences (276 from receptionists, 224 from customers). This corpus was further expanded to 1,677 sentences by incorporating variants, and translated into LSE (both text glosses and video). The corpus statistics are detailed in Table 1, categorizing sentences by speaker (receptionist/customer) and providing counts for sentence pairs, different sentences, running words, and vocabulary.
Key Insights and Ideas
- User-Centricity is Paramount: The success of domain adaptation hinges on actively involving end-users (deaf customers and receptionists) in defining requirements and evaluating the system. This ensures the final product is truly usable and meets their needs.
- Parallel Corpus is the Engine: A well-curated, domain-specific parallel corpus is the linchpin for adapting both speech recognition and translation modules. The quality and quantity of this corpus directly impact the system's performance.
- Hybrid Translation for Robustness: Combining example-based translation with a statistical background module provides a robust approach. The example-based method handles common patterns efficiently, while the statistical module offers a fallback for less frequent or more complex translations.
- Generalization Enhances Adaptability: Generalizing translation examples by replacing specific words with categories (e.g., numbers, months) significantly increases the number of sentences that can be translated effectively from a limited set of examples.
- LSESpeak's Dual Utility: The integration of LSESpeak, offering both LSE-to-Spanish and SMS-to-Spanish translation, addresses the bidirectional communication needs and acknowledges the prevalence of SMS language among deaf users.
Key Results
The adapted system demonstrated strong performance in the hotel reception domain:
- Speech Recognition: Achieved a Word Error Rate (WER) of 6.7%.
- Speech-to-Sign Translation: Resulted in a Sign Error Rate (SER) of 10.7%.
- Dialogue Speed: The total time for translating speech into LSE (recognition + translation + sign representation) was around 7 seconds, enabling agile dialogue.
- Spanish Generation: The spoken Spanish generator from LSE achieved a translation rate of 98.0% with an average translation time of 0.001 seconds and text-to-speech conversion taking 2.1 seconds.
- User Specification Time: Deaf users took an average of 18.0 seconds to specify a gloss sequence using the interface, which is considered reasonable given limited practice time.
- Subjective Evaluation: For isolated signs, human recognition rate was high (87.5% on the first attempt). For questions within dialogues, the rate was lower (62.5% on the first attempt), attributed to regional variations in LSE and sign representation nuances.
Limitations and Future Work
- Sequential Methodology: The paper notes that the methodology is sequential, meaning the technology adaptation is dependent on the parallel corpus generation. This could potentially lead to bottlenecks.
- LSE Variability: The authors acknowledge that LSE is a relatively young language with significant regional variability, which impacts sign representation and can affect recognition accuracy.
- Future Work: While not explicitly detailed, the results suggest that further refinement of the LSE representation and potentially more advanced adaptation techniques could improve performance, especially in handling the nuances of sign language variations. The authors also mention the need for more practice time for users to reduce gloss sequence specification time.
Summary updated