Development of a Mandarin-English Bilingual Speech Recognition System for Real World Music Retrieval

Qingqing Zhang, Jielin Pan, Yang Lin, Jian Shao, Yonghong Yan

Abstract

This paper presents a robust Mandarin-English bilingual speech recognition system (MESRS) specifically optimized for real-world music retrieval tasks. The system addresses two primary challenges: the computational complexity of handling intra-sentential code-switching and the performance degradation caused by non-native accents. By employing a novel "Two-pass Confusion Matrix" (TCM) for data-driven phone clustering and a strategic model retraining approach, the researchers achieved a significant reduction in Phrase Error Rates (PER)—approximately 24% for English queries—without compromising the performance of the native Mandarin "matrix" language.

Introduction

In many multilingual societies, "code-switching"—the practice of alternating between two or more languages within a single conversation or sentence—is a common linguistic phenomenon. In the context of music retrieval, users frequently mix languages, such as saying, "One night in 北京, 阿ben." Developing a speech recognition system that can handle such inputs seamlessly is difficult for two main reasons.

First, traditional approaches often rely on two separate monolingual engines, which is computationally expensive and fails to handle transitions within a single sentence effectively. Second, there is a significant "accent gap." Most bilingual users have a native language (in this case, Mandarin) and an embedded language (English). Their English often carries heavy accents that standard acoustic models, typically trained on native English speakers, struggle to recognize.

Methodology

Two-pass Confusion Matrix (TCM) Phone Clustering

To streamline the system, the researchers proposed a unified acoustic model using a merged set of bilingual phones. Rather than relying on linguistic theory alone, they developed the Two-pass Confusion Matrix (TCM), a data-driven approach to phone clustering.

Figure 1: Illustration of the Two-pass Confusion Matrix (TCM) phone clustering process, showing the alignment between target reference and source hypothesis phones across languages.

As shown in Figure 1, the TCM process involves force-aligning speech from a target language (e.g., Mandarin) and running a recognizer from the source language (e.g., English) over it. By analyzing which sounds are frequently confused across the two languages, the system can cluster similar phones into a compact set of 70 classes. This reduces the total number of parameters, making the model more efficient and easier to train with limited bilingual data.

Non-Native Adaptation Strategies

To bridge the accent gap for Mandarin-accented English, the study compared two adaptation methods:

  1. Maximum A Posteriori (MAP): A standard technique that shifts model parameters toward new data.
  2. Model Retraining: Appending non-native English data to the original training set and retraining the model from scratch.

The researchers found that while MAP is effective for monolingual adaptation, it can "pull" a bilingual model too far away from its native Mandarin roots. Retraining emerged as a superior compromise, broadening the phonetic coverage of the model to include accented English while maintaining the integrity of the Mandarin acoustic space.

Experimental Setup

The system was trained using a combination of the DB863 Mandarin corpus, the Wall Street Journal (WSJ) native English corpus, and the GTC corpus (24,000 English utterances by Mandarin speakers). Evaluation was conducted on a custom music retrieval dataset featuring over 10,000 utterances recorded in challenging, noisy environments like streets and restaurants to simulate real-world usage.

Results and Discussion

The final MESRS system demonstrated substantial improvements over both monolingual baselines and simple combined bilingual models.

Figure 2: Comparison of Phrase Error Rates (PER) across different models (Mandarin-only, English-only, Combined, and MESRS) on Mandarin (Test M), English (Test E), and Bilingual (Test B) datasets.

As illustrated in Figure 2, the MESRS system significantly outperformed the baseline models, particularly in the English and Bilingual test sets:

  • English Utterances: Achieved a 24.47% relative reduction in PER compared to a native English baseline.
  • Bilingual Utterances: Achieved a 22.37% relative reduction in PER.
  • Mandarin Utterances: Maintained performance levels comparable to the specialized Mandarin-only system, proving that bilingual optimization does not necessitate a trade-off in native language accuracy.

Conclusion

The development of the MESRS highlights the power of data-driven clustering and balanced retraining in creating robust bilingual speech interfaces. While currently optimized for the music retrieval domain, the TCM and retraining methodologies offer a scalable framework for other code-switching applications. Future research will focus on expanding these techniques to open-domain, large-vocabulary tasks and exploring larger non-native datasets to further refine the adaptation process.

Summary updated

BibTeX
@article{zhang2008development,
  author = {Qingqing Zhang and Jielin Pan and Yang Lin and Jian Shao and Yonghong Yan},
  title = {{Development of a Mandarin-English Bilingual Speech Recognition System for Real World Music Retrieval}},
  journal = {IEICE Trans. Information and Systems},
  year = {2008},
  doi = {10.1093/ietisy/e91-d.3.514},
}
Figure