Abstract
This paper presents a robust Mandarin-English bilingual speech recognition system (MESRS) specifically optimized for real-world music retrieval tasks. The system addresses two primary challenges: the computational complexity of handling intra-sentential code-switching and the performance degradation caused by non-native accents. By employing a novel "Two-pass Confusion Matrix" (TCM) for data-driven phone clustering and a strategic model retraining approach, the researchers achieved a significant reduction in Phrase Error Rates (PER)—approximately 24% for English queries—without compromising the performance of the native Mandarin "matrix" language.
Introduction
In many multilingual societies, "code-switching"—the practice of alternating between two or more languages within a single conversation or sentence—is a common linguistic phenomenon. In the context of music retrieval, users frequently mix languages, such as saying, "One night in 北京, 阿ben." Developing a speech recognition system that can handle such inputs seamlessly is difficult for two main reasons.
First, traditional approaches often rely on two separate monolingual engines, which is computationally expensive and fails to handle transitions within a single sentence effectively. Second, there is a significant "accent gap." Most bilingual users have a native language (in this case, Mandarin) and an embedded language (English). Their English often carries heavy accents that standard acoustic models, typically trained on native English speakers, struggle to recognize.
Methodology
Two-pass Confusion Matrix (TCM) Phone Clustering
To streamline the system, the researchers proposed a unified acoustic model using a merged set of bilingual phones. Rather than relying on linguistic theory alone, they developed the Two-pass Confusion Matrix (TCM), a data-driven approach to phone clustering.
As shown in Figure 1, the TCM process involves force-aligning speech from a target language (e.g., Mandarin) and running a recognizer from the source language (e.g., English) over it. By analyzing which sounds are frequently confused across the two languages, the system can cluster similar phones into a compact set of 70 classes. This reduces the total number of parameters, making the model more efficient and easier to train with limited bilingual data.
Non-Native Adaptation Strategies
To bridge the accent gap for Mandarin-accented English, the study compared two adaptation methods:
- Maximum A Posteriori (MAP): A standard technique that shifts model parameters toward new data.
- Model Retraining: Appending non-native English data to the original training set and retraining the model from scratch.
The researchers found that while MAP is effective for monolingual adaptation, it can "pull" a bilingual model too far away from its native Mandarin roots. Retraining emerged as a superior compromise, broadening the phonetic coverage of the model to include accented English while maintaining the integrity of the Mandarin acoustic space.
Experimental Setup
The system was trained using a combination of the DB863 Mandarin corpus, the Wall Street Journal (WSJ) native English corpus, and the GTC corpus (24,000 English utterances by Mandarin speakers). Evaluation was conducted on a custom music retrieval dataset featuring over 10,000 utterances recorded in challenging, noisy environments like streets and restaurants to simulate real-world usage.
Results and Discussion
The final MESRS system demonstrated substantial improvements over both monolingual baselines and simple combined bilingual models.
As illustrated in Figure 2, the MESRS system significantly outperformed the baseline models, particularly in the English and Bilingual test sets:
- English Utterances: Achieved a 24.47% relative reduction in PER compared to a native English baseline.
- Bilingual Utterances: Achieved a 22.37% relative reduction in PER.
- Mandarin Utterances: Maintained performance levels comparable to the specialized Mandarin-only system, proving that bilingual optimization does not necessitate a trade-off in native language accuracy.
Conclusion
The development of the MESRS highlights the power of data-driven clustering and balanced retraining in creating robust bilingual speech interfaces. While currently optimized for the music retrieval domain, the TCM and retraining methodologies offer a scalable framework for other code-switching applications. Future research will focus on expanding these techniques to open-domain, large-vocabulary tasks and exploring larger non-native datasets to further refine the adaptation process.
Summary updated