Speechalator: Two-Way Speech-to-Speech Translation in Your Hand

Alex Waibel, Ahmed Badran, Alan W. Black, Robert E. Frederking, Donna Gates, Alon Lavie, Lori S. Levin, Kevin A. Lenzo, Laura Mayfield Tomokiyo, Jürgen Reichert, Tanja Schultz, Dorcas Wallace, Monika Woszczyna, Jing Zhang

Abstract

The Speechalator represents a significant milestone in mobile computing, being one of the first systems to integrate a complete, two-way speech-to-speech translation pipeline onto a consumer-grade Personal Digital Assistant (PDA). Developed in 2003 as part of the DARPA-funded Babylon project, it enabled real-time communication between English and Egyptian Arabic speakers within the medical domain, overcoming severe hardware constraints and linguistic complexities.

Introduction

In the early 2000s, mobile translation technology was primarily limited to "one-way" devices. These tools, such as the Phrasalator, could recognize a set of pre-defined phrases and play back pre-recorded translations but lacked the flexibility required for natural, back-and-forth dialogue. The Speechalator was designed to bridge this gap by providing a fully automated, two-way pipeline—comprising Automatic Speech Recognition (ASR), Machine Translation (MT), and Text-to-Speech (TTS)—on a device with a 200–400MHz processor and as little as 32MB of RAM.

System Architecture and Optimization

To operate within the strict resource limits of a Pocket PC, the researchers employed a modular pipeline optimized for efficiency. A key innovation was the use of an "interlingua" approach to ensure scalability and reduce computational load.

Recognition and Translation

The system utilized an HMM-based recognizer from Multimodal Technologies Inc. (MTI). To minimize computational overhead, the grammar was tightly coupled with the recognizer. Crucially, the ASR was modified to output an Interlingua Interchange Format (IF) directly. This "shortcut" eliminated the need for a separate natural language parsing module on the handheld device, significantly reducing the memory footprint.

For translation, the team implemented a statistical training method that mapped structured IF to target text. While traditional knowledge-based engines proved too resource-intensive for local execution, this statistical model was lean enough to run on the PDA without requiring a wireless link for basic functionality.

The Speechalator interface on a Pocket PC, showing a translation from romanized Egyptian Arabic to English.

Synthesis

Speech synthesis was handled by Cepstral’s Theta system. To fit within the memory constraints, the team used optimized 11KHz voices, reducing the storage requirement from hundreds of megabytes to just 9MB per voice.

Linguistic Challenges: Egyptian Arabic

The choice of Egyptian Arabic over Modern Standard Arabic (MSA) was a strategic decision. While MSA is the academic and formal standard, Egyptian Arabic is the primary spoken dialect, making it more practical for real-world field communication.

Because written Arabic typically omits vowels, the team adopted an internal romanization scheme based on the CallHome project. As seen in the system interface, this phonetic representation served as the essential bridge between the ASR, the translation engine, and the TTS components.

Performance and Results

The Speechalator achieved an end-to-end latency of 2 to 4 seconds, measured from the end of speech to the start of the translated audio output. In informal testing within the medical interview domain, the system demonstrated an in-domain accuracy of over 80%.

Interestingly, the researchers noted a hardware anomaly: the older StrongARM 206MHz processors performed slightly faster than the newer XScale 400MHz processors. This was attributed to a lack of software optimization for the newer architecture at the time of development.

Conclusion and Future Work

The Speechalator proved that complex, multi-stage speech processing could be localized to mobile hardware. However, limitations remain, including the system's strict domain specificity (medical interviews) and the need for more formal, large-scale evaluations. Future work aims to expand the vocabulary for general conversation and further optimize the engines for modern mobile architectures.

Summary updated

Abstract

This demonstration involves two-way automatic speechto-speech translation on a consumer off-the-shelf PDA. This work was done as part of the DARPA-funded Babylon project, investigating better speech-to-speech translation systems for communication in the field. The development of the Speechalator software-based translation system required addressing a number of hard issues, including a new language for the team (Egyptian Arabic), close integration on a small device, computational efficiency on a limited platform, and scalable coverage for the domain.
BibTeX
@inproceedings{waibel2003speechalator,
  author = {Alex Waibel and Ahmed Badran and Alan W. Black and Robert E. Frederking and Donna Gates and Alon Lavie and Lori S. Levin and Kevin A. Lenzo and Laura Mayfield Tomokiyo and Jürgen Reichert and Tanja Schultz and Dorcas Wallace and Monika Woszczyna and Jing Zhang},
  title = {{Speechalator: Two-Way Speech-to-Speech Translation in Your Hand}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
}
Figure