Search by voice in Mandarin Chinese

Jiulong Shan, Genqing Wu, Zhihong Hu, Xiliu Tang, Martin Jansche, Pedro J. MorenoInterspeech 2010

TL;DR

This paper details the development of Google's first Mandarin Chinese voice search system, tackling unique challenges like non-Latin scripts and word segmentation. The system achieved excellent performance on real-world queries and has been successfully deployed, serving as a template for future internationalization efforts.

What Problem Does This Paper Address?

While voice search systems for English have seen significant development, other languages, particularly those in East and Southeast Asia, present unique hurdles. Mandarin Chinese, with its lack of explicit word boundaries in writing and its tonal nature, requires specialized approaches for speech recognition. The authors aimed to build a robust Mandarin voice search system, addressing challenges in data collection, language and acoustic modeling, and text normalization, to make voice search accessible to a wider global audience.

Methods

The authors employed a multi-faceted approach:

  • Data Collection: A client/server application on Android phones was used to collect over 250,000 utterances from more than 1200 speakers. This data was collected under varied acoustic conditions and included queries with mixed Mandarin and English terms.
  • Language Modeling: An n-gram language model was built using typed web queries from google.cn. This involved custom segmentation, filtering of irrelevant terms (e.g., very long queries, URLs, offensive content), and normalization of URLs. They explored 3-, 4-, and 5-gram models, settling on a 4-gram model for its balance of performance and complexity.
  • Lexicon Development: A significant challenge was creating pronunciations for a large, word-based vocabulary. They combined a small static dictionary of common words with a rule-based system for Chinese characters. Foreign words were handled by looking them up in an English pronunciation dictionary and mapping to Mandarin phones. Non-standard words, numbers, and alphanumeric sequences were also addressed.
  • Acoustic Modeling: Standard 3-state context-dependent (triphone) models were used, trained on PLP cepstral coefficients. A key innovation was handling Mandarin's tonal nature by creating a phoneme inventory that replicated vowels and diphthongs for each of the four main tones, effectively modeling tones indirectly. They used gender-independent, maximum-likelihood trained models, followed by boosted MMI.
  • Interpolated Language Models: To further improve performance, they incorporated a language model built from automatically transcribed recognition hypotheses, interpolated with their primary query-based model.

Datasets

  • Training Data:
    • Initial corpus: ~250,000 transcribed utterances (TrainSet1).
  • Augmented corpus: ~750,000 transcribed utterances (TrainSet2).
  • Further augmented corpus: ~1 million transcribed utterances (TrainSet3).
  • Production data: Over 500,000 additional transcribed utterances.
  • Unsupervised data: ~350,000 automatically transcribed queries for language model interpolation.
  • Testing Data:
    • A set of ~4,000 queries from real users, containing only Mandarin utterances.
  • A smaller test subset of ~2,000 utterances from speakers with Cantonese accents.

Key Insights and Ideas

  • Voice Search is Different from Dictation: The authors highlight that voice search queries are often shorter, more varied, and less formal than dictated text, requiring different modeling strategies.
  • User Comfort with Spoken English: Surprisingly, users who readily typed English queries struggled to speak them, informing the language modeling approach.
  • Tonal Modeling Indirectly: Instead of directly modeling pitch, replicating phonemes for each tone proved an effective way to handle Mandarin's tonal distinctions.
  • Value of Production Data: Augmenting training data with real-world production utterances led to significant performance gains, demonstrating the importance of large-scale, domain-specific data.
  • Unsupervised Data for LM Improvement: Using recognition hypotheses to build language models offered further performance boosts, suggesting a path for continuous improvement.

Key Results

  • Performance Metrics: The system achieved a Character Error Rate (CER) of 23.1% and a Web Score at 1 (WSC@1) of 63.3% using TrainSet3 and a 4-gram language model.
  • Data Impact: Increasing acoustic training data from 250k to 1 million utterances led to an absolute CER reduction of nearly 12 percentage points and a WSC@1 increase of over 13 percentage points.
  • Language Model Interpolation: Interpolating with an unsupervised LM further improved CER to 21.0% and WSC@1 to 65.6%.
  • Dialectal Performance: Utterances from Cantonese-accented speakers showed a degradation in performance (CER up to 27.0%, WSC@1 down to 62.3%), indicating a need for acoustic adaptation.
  • System Deployment: The system has been in operation since October 2009 and has received very positive user reviews.

Limitations and Future Work

  • Dialectal Adaptation: The performance degradation on Cantonese-accented Mandarin highlights the need for acoustic adaptation techniques to handle regional dialectal variations.
  • Saturation Point: The authors suggest that current acoustic and language modeling techniques may not have reached a saturation point, implying further improvements are possible with more data and refined methods.
  • Template for Internationalization: The system serves as a successful template for internationalizing voice search to other languages, a key direction for future work.

Summary updated

BibTeX
@inproceedings{shan2010search,
  author = {Jiulong Shan and Genqing Wu and Zhihong Hu and Xiliu Tang and Martin Jansche and Pedro J. Moreno},
  title = {{Search by voice in Mandarin Chinese}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure