A hybrid phrase-based/statistical speech translation system

David Stallard, Fred Choi, Kriste Krstovski, Prem Natarajan, Rohit Prasad, Shirin SaleemInterspeech 2006

TL;DR

This paper introduces a hybrid speech translation system designed for two-way communication between English and colloquial Iraqi Arabic, specifically for military applications like force protection. It cleverly combines the reliability of pre-recorded phrases for common queries with the flexibility of statistical machine translation for more open-ended conversations, aiming to bridge the gap between clarity and broad coverage.

What Problem Does This Paper Address?

Effective communication between military personnel and local populations in foreign countries is crucial, especially in crisis situations where language barriers can be significant. Traditional approaches either rely on extensive human interpreter resources, which are often scarce, or on fully automated systems that can suffer from low intelligibility and uncertainty in translation accuracy. This paper tackles the challenge of building a practical, robust speech translation system for a low-resource language (colloquial Iraqi Arabic) that balances the need for clear, reliable communication for common tasks with the ability to handle more diverse conversations.

Methods

The system employs a hybrid architecture that leverages the strengths of two distinct approaches:

  • Task-Directed (1.5-way) Approach: For common, predictable queries (e.g., "What is the color of the car?"), the system uses a "Question Canonicalizer." This module maps incoming English speech recognition output to a predefined set of ~700 canonical questions for which pre-recorded Arabic translations exist. This ensures high accuracy and intelligibility for these critical interactions.
  • Statistical Machine Translation (SMT) (2-way) Approach: When a query doesn't match a canonical question, or when multiple matches are found, the system falls back to a phrase-based SMT engine. This engine translates English text to Iraqi Arabic text, which is then synthesized into speech. For the Arabic-to-English direction, it combines SMT with an information extraction component to capture relevant details.
  • Speech Recognition: Both English and Arabic speech are recognized using BBN's Byblos system, which employs context-dependent phonetic Hidden Markov Models (HMMs) with tied Gaussian mixtures. Discriminative training (Maximum Mutual Information - MMI) is used to improve acoustic models over baseline Maximum Likelihood (ML) training.
  • Speech Synthesis: For English-to-Arabic translation, a speech synthesizer is used. For Arabic-to-English, a Cepstral Iraqi speech synthesizer is employed.

Datasets

  • Iraqi Arabic Speech Recognition: Trained on approximately 196 hours of Iraqi Arabic speech collected under the TRANSTAC effort, encompassing various domains (Medical/Refugee, Force Protection, Civic Amenities, Detainee Screening).
  • English Speech Recognition: Trained on 36 hours of English speech.
  • Statistical Machine Translation:
    • Arabic-to-English (A2E): Developed using ~236K parallel utterances.
  • English-to-Arabic (E2A): Developed using ~184K parallel utterances, augmented with the reverse of A2E data.
  • Question Canonicalizer: Trained on a corpus of approximately 70,000 variants of canonical utterances.

Key Insights and Ideas

  • Hybridization is Key: The core idea is that a purely SMT system can be too uncertain for critical military interactions, while a purely task-directed system is too limited. Combining them offers the best of both worlds: reliability for common tasks and flexibility for the unexpected.
  • Low-Resource Language Challenges: Colloquial Iraqi Arabic is treated as a low-resource language, highlighting the difficulties in obtaining large parallel corpora and the potential for synthesized speech to be less intelligible.
  • Clarity vs. Coverage Trade-off: The paper explicitly discusses the trade-off between the clarity and guaranteed accuracy of the 1.5-way approach and the broader coverage of the 2-way SMT approach.
  • User Interface Design: The use of a push-and-hold interface for speech input is noted as being more robust and familiar for military users than automatic end-of-speech detection.

Key Results

  • Speech Recognition:
    • Iraqi Arabic: MMI training improved WER by 11.5% relative over ML. Adapted models achieved ~22.6% WER on the 1.5-way offline test set.
  • English: MMI training yielded an 11% relative gain in WER. Adapted models achieved ~5.80% WER on the 1.5-way offline test set.
  • Translation:
    • Arabic-to-English (A2E): Achieved a BLEU score of 0.54 with the December 2005 configuration and 0.61 with the March 2006 configuration. Speech output BLEU improved from 0.43 to 0.54.
  • English-to-Arabic (E2A): Performance was notably poorer than A2E, with BLEU scores of 0.32 (Dec-05) and 0.43 (Mar-06). The paper attributes this to less available data and the morphological complexity of Arabic.
  • Utterance Canonicalization: Achieved error rates of 1.3% on transcription and 6.9% on ASR output for the 1.5-way task.

Limitations and Future Work

  • Limitations:
    • The English-to-Arabic translation performance is significantly lower than Arabic-to-English, attributed to data scarcity and linguistic differences.
  • The system currently runs on a Windows laptop, with a port to a handheld device mostly complete but lacking the SMT engine.
  • Future Work:
    • Porting the SMT component to the handheld environment.
  • Continued improvement and testing of the system.

Summary updated

BibTeX
@inproceedings{stallard2006hybrid,
  author = {David Stallard and Fred Choi and Kriste Krstovski and Prem Natarajan and Rohit Prasad and Shirin Saleem},
  title = {{A hybrid phrase-based/statistical speech translation system}},
  booktitle = {Interspeech},
  year = {2006},
}
Figure