A grammar-based Chinese to English speech translation system for portable devices

Pascale Fung, Yi Liu, Yongsheng Yang, Yihai Shen, Dekai WuInterspeech 2004 · DOI

In 2004, researchers at the Hong Kong University of Science & Technology (HKUST) presented a groundbreaking approach to real-time speech translation on mobile devices. Long before the ubiquity of high-performance smartphone GPUs and cloud-based AI, this system managed to fit a complete Chinese-to-English translation pipeline into a mere 500KB of memory, operating efficiently on 200MHz CPUs.

System Architecture

The primary challenge addressed by this research was the severe hardware constraints of early 2000s portable devices, such as PDAs. These devices lacked floating-point units and had very limited RAM, making standard Large Vocabulary Continuous Speech Recognition (LVCSR) and statistical machine translation (SMT) models impractical.

The authors developed a domain-specific system (e.g., for travel or hotel bookings) that was computationally "thin" yet linguistically robust. The high-level architecture consists of an online front-end process, a speech decoder guided by a Context-Free Grammar (CFG), and a translation module utilizing Unambiguous Inversion Transduction Grammars (ITGs).

System architecture of the speech-to-translation pipeline.

Optimizing the Front-End for Mobile Hardware

One of the most significant performance bottlenecks in speech processing is the extraction of Mel-frequency cepstral coefficients (MFCCs). Standard implementations rely heavily on floating-point arithmetic, which is slow to emulate on mobile processors.

To overcome this, the researchers implemented a fixed-point front-end. They replaced standard decimal math with scaled integers and look-up tables for complex trigonometric and logarithmic functions. This optimization reduced the computation time for speech features from 2100ms to just 80ms—a 26x speedup that was crucial for achieving real-time performance.

Detailed flow of the front-end processing module, optimized for fixed-point computation.

Efficient Acoustic Modeling and Decoding

The system employs several strategies to balance accuracy and memory usage:

  • Class-Based Bi-phone Models: Instead of heavy triphone models or overly simplistic monophones, the authors used "bi-phones." By grouping Mandarin initials and finals into classes based on phonetic similarity, they reduced the acoustic model's memory footprint from over 2MB to less than 400KB, while maintaining high accuracy.
  • CFG Decoding with Attributes: The speech decoder uses a CFG that incorporates "attributes." This allows the system to semantically distinguish between similar-sounding strings, such as identifying whether a sequence of numbers represents a code or a numerical value.

Linear Time Translation with ITGs

For the translation step, the system utilizes Inversion Transduction Grammars (ITGs). A key innovation was the use of "unambiguous" ITGs. By constraining the grammar rules to be unambiguous, the computational complexity of the translation process was reduced from $O(n^3)$ to linear time $O(n)$. This efficiency allowed the translation to occur without perceptible lag on low-power 200MHz processors.

Key Results and Impact

The optimized system demonstrated that sophisticated speech translation was possible on highly constrained hardware:

  • Memory Footprint: The entire pipeline required only 500KB of RAM.
  • Speed: The 26x speedup in front-end processing enabled real-time interaction.
  • Accuracy: Class-based bi-phone models provided a 10% accuracy increase over standard monophone models.

While the system was domain-specific and required manually authored grammar rules, it proved that strategic engineering—such as fixed-point math and grammar constraints—could bridge the gap between desktop-class algorithms and mobile hardware realities. This work laid the groundwork for the portable translation tools we use today.

Summary updated

BibTeX
@inproceedings{fung2004grammar,
  author = {Pascale Fung and Yi Liu and Yongsheng Yang and Yihai Shen and Dekai Wu},
  title = {{A grammar-based Chinese to English speech translation system for portable devices}},
  booktitle = {Interspeech},
  year = {2004},
  doi = {10.21437/interspeech.2004-40},
}
Figure