Development of phrase translation systems for handheld computers: from concept to field

Horacio Franco, Jing Zheng, Kristin Precoda, Federico Cesari, Victor Abrash, Dimitra Vergyri, Anand Venkataraman, Harry Bratt, Colleen Richey, Ace SarichInterspeech 2003 · DOI

TL;DR

This paper introduces the development of spoken phrase translation systems (SPTS) for handheld devices, starting with unidirectional English translation and evolving to a bidirectional English-Pashto system. The work addresses the challenges of real-time speech recognition and translation on resource-constrained platforms, particularly for naive users in asymmetric communication scenarios.

What Problem Does This Paper Address?

Traditional phrasebooks offer a basic way to bridge language gaps, but they are static and require users to manually search for phrases. This paper tackles the challenge of creating dynamic, voice-activated translation systems that can run on handheld computers, a significant technical hurdle due to limited processing power and memory. The authors specifically address the difficulties of enabling even limited bidirectional communication, especially when dealing with users unfamiliar with the system's capabilities or expected responses. This is crucial for applications like disaster relief, medical assistance, and peacekeeping where quick, essential communication is vital.

Methods

The core of the system relies on SRI's DynaSpeak speech recognition engine, optimized for embedded and mobile systems. Key methodological contributions include:

  • Speaker-Independent Recognition: Enabling immediate use without individual speaker training.
  • Phonetically-Based Acoustic Models: Facilitating easy addition of new phrases through simple text input and recordings.
  • Grammar Management: Utilizing JSGF format for defining recognition grammars and supporting dynamically loadable grammars.
  • Unidirectional SPTS Architecture: A straightforward lookup mechanism where recognized English phrases trigger corresponding prerecorded audio translations.
  • Bidirectional SPTS Architecture: Extending the unidirectional system by implementing two SPTSs (English and Pashto). A crucial innovation here is the question-dependent recognition grammar for the respondent's language (Pashto). This means the Pashto recognition grammar dynamically adapts based on the last English question, significantly reducing recognition perplexity by constraining possible answers to semantically similar classes.
  • Wordspotting Capability: To handle unexpected but potentially critical input from naive respondents, a wordspotting mechanism was integrated. This uses a class-based bigram language model and explicit acoustic modeling to detect "hot words" within a limited real-time footprint.
  • Structured Value Sentences: A mechanism to parse and translate sentences containing quantities, time, or order information by embedding these values within grammars and dynamically concatenating prerecorded waveforms.
  • Online Adaptation: Fast online speaker and environment adaptation for improved recognition robustness.

Datasets

Not explicitly detailed in the paper. The authors mention the use of "English acoustic models" and "Pashto language technology," implying existing or developed resources for these languages. The system is designed to be extensible, allowing for the addition of new languages and domains with recorded waveforms.

Key Insights and Ideas

  • The Power of Asymmetry: The SPTS approach is particularly effective in communication scenarios where one party primarily directs the conversation (e.g., giving instructions, asking simple questions), making it ideal for asymmetric interactions.
  • Question-Dependent Grammars are Key: For bidirectional communication with naive users, dynamically adapting recognition grammars based on the preceding question is essential for managing complexity and achieving usable accuracy on limited hardware.
  • Wordspotting for Robustness: Integrating wordspotting is vital for handling unexpected but important utterances from respondents, preventing complete communication breakdown.
  • Structured Value Handling: The ability to parse and translate structured information (like numbers or dates) dynamically adds significant practical utility beyond simple phrase translation.
  • Rapid Prototyping: The system's design, particularly the ease of adding new phrases, domains, and languages, makes it a valuable tool for rapid development of specialized translation packages.

Key Results

  • The unidirectional SPTS was successfully deployed and used in real-world situations, providing valuable feedback on its utility.
  • The bidirectional SPTS was developed to handle English-Pashto communication, incorporating question-dependent grammars and wordspotting.
  • The wordspotting algorithm demonstrated a significant reduction in false alarms compared to classical methods.
  • Field experience indicated that users quickly adapted to the SPTS limitations and expressed a strong desire for bidirectional capabilities.
  • The system's architecture is designed to be computationally moderate, allowing it to run on current handheld computers.

Limitations and Future Work

  • Naive Respondent Challenges: The paper acknowledges that even with question-dependent grammars and wordspotting, dealing with completely untrained respondents remains a challenge.
  • Phrasebook Limitations: The SPTS is fundamentally a phrase translation system and cannot handle spontaneous, free-form speech. Users must adapt to the system's constraints.
  • User Interface and Phrase Organization: Feedback suggests that users would benefit from more flexible ways to organize and manage phrases, potentially allowing for user-defined domains and hierarchies.
  • Future Work: The authors suggest further development in areas like user-defined phrase organization and potentially more sophisticated handling of spontaneous speech, though the core SPTS approach remains focused on constrained communication.

Summary updated

BibTeX
@inproceedings{franco2003development,
  author = {Horacio Franco and Jing Zheng and Kristin Precoda and Federico Cesari and Victor Abrash and Dimitra Vergyri and Anand Venkataraman and Harry Bratt and Colleen Richey and Ace Sarich},
  title = {{Development of phrase translation systems for handheld computers: from concept to field}},
  booktitle = {Interspeech},
  year = {2003},
  doi = {10.21437/eurospeech.2003-148},
}
Figure