Advances in Mandarin broadcast speech recognition

Mei-Yuh Hwang, Wen Wang, Xin Lei, Jing Zheng, Özgür Çetin, Gang Peng

Researchers from the University of Washington (UW), SRI International, and ICSI have achieved a significant breakthrough in Mandarin broadcast speech recognition. By scaling training data and implementing a sophisticated multi-stage decoding pipeline, they achieved a relative reduction in Character Error Rate (CER) ranging from 24% to 64% across various benchmarks.

The Challenge of Broadcast Media

Automatic Speech Recognition (ASR) for Mandarin broadcast media faces two primary challenges: the diversity of the "Broadcast News" (BN) genre and the inherent difficulty of "Broadcast Conversation" (BC). While BN consists of planned, clear speech, BC involves spontaneous talk shows and call-in segments characterized by disfluencies and informal language.

The authors sought to bridge the performance gap between these genres. They addressed limitations in previous systems, such as small training sets (only 97 hours of audio), limited lexicons, and simple decoding architectures that didn't fully exploit the differences between MFCC-based and neural-network-based acoustic features.

A Multi-Stage Decoding Architecture

The paper describes a comprehensive overhaul of the ASR pipeline, focusing on feature diversity and iterative refinement. The system uses two distinct front-ends: standard MFCCs plus pitch features, and Multi-Layer Perceptron (MLP) phoneme posteriors reduced via Principal Component Analysis (PCA).

The multi-stage decoding pipeline featuring cross-adaptation between MLP and MFCC models.

A key architectural innovation is Cross-Adaptation Decoding. Instead of a single pass, the system uses two iterations of cross-adaptation. An initial pass with an MLP-based model provides a hypothesis used to adapt the MFCC-based model, and vice versa. This strategy is more effective than self-adaptation because the two models make different types of errors, allowing them to "teach" each other during the decoding process.

Additional methods included:

  • Acoustic Training: State-of-the-art discriminative training criteria, specifically Minimum Phone Error (MPE) and feature-space MPE (fMPE), were used to minimize expected error rates.
  • Language Model (LM) Hierarchy: The vocabulary was expanded from 49K to 60K words, utilizing a suite of five static LMs ranging from "quick" bigrams for fast first-pass decoding to massive 5-grams for final rescoring.
  • Unsupervised LM Adaptation: For the difficult BC genre, a dynamic interpolation method adjusted the LM on a per-show basis using first-pass decoding results.

Scaling Data for Robustness

The researchers significantly scaled their data compared to prior work, particularly in the acoustic domain. The acoustic training data was increased from 97 to approximately 465 hours, incorporating a substantial amount of the challenging Broadcast Conversation genre.

Summary of acoustic training data sources and hours for Broadcast News (BN) and Broadcast Conversation (BC).

Text data for Language Model training also saw a massive increase, with over 1 billion words sourced from the Gigaword corpus, web-crawled data, and Conversational Telephone Speech (CTS) transcripts.

Key Results and Insights

The improvements across all benchmarks were substantial:

  • CER Reductions: On the ext06 test set, the CER dropped from 15.0% to 5.4% (a 64% relative improvement).
  • Benchmark Performance: On the standard dev04 set, the system achieved a CER of 3.7%, down from 6.0%.
  • BC Genre Improvement: On the difficult dev05bc set, the system improved from 34.0% to 20.6% CER.
  • LM Adaptation: Specifically for conversations, unsupervised LM adaptation provided an additional 6% relative CER reduction.

The study highlights that combining MLP-based phoneme posteriors with traditional MFCCs provides a "superset" of features that captures both short-term spectral information and longer-term phonetic context.

Conclusion and Future Work

Despite these advancements, a significant performance gap remains between Broadcast News (~5% CER) and Broadcast Conversation (~20% CER), confirming that spontaneous speech remains a difficult problem. Furthermore, the multi-pass cross-adaptation and 5-gram rescoring architecture is computationally expensive, presenting challenges for real-time applications. Future research will explore more granular per-topic dynamic interpolation and Maximum A Posteriori (MAP) learning for LM adaptation to further refine performance on spontaneous speech.

Summary updated

BibTeX
@inproceedings{hwang2007advances,
  author = {Mei-Yuh Hwang and Wen Wang and Xin Lei and Jing Zheng and Özgür Çetin and Gang Peng},
  title = {{Advances in Mandarin broadcast speech recognition}},
  booktitle = {Interspeech},
  year = {2007},
  doi = {10.21437/interspeech.2007-589},
}
Figure