Rnn-transducer with language bias for end-to-end Mandarin-English code-switching speech recognition

Shuai Zhang, Jiangyan Yi, Zhengkun Tian, Jianhua Tao, Ye Bai

Abstract

Recently, language identity information has been utilized to improve the performance of end-to-end code-switching (CS) speech recognition. However, previous works use an additional language identification (LID) model as an auxiliary module, which causes the system complex. In this work, we propose an improved recurrent neural network transducer (RNN-T) model with language bias to alleviate the problem. We use the language identities to bias the model to predict the CS points. This promotes the model to learn the language identity information directly from transcription, and no additional LID model is needed. We evaluate the approach on a Mandarin-English CS corpus SEAME. Compared to our RNN-T baseline, the proposed method can achieve 16.2% and 12.9% relative error reduction on two test sets, respectively.

Introduction

Code-switching (CS)—the practice of alternating between two or more languages in a single conversation—is a major hurdle for Automatic Speech Recognition (ASR). Standard systems often struggle at "switch points," where phonetic and linguistic rules shift abruptly. Historically, researchers addressed this by adding separate Language Identification (LID) modules, which increased system complexity and latency.

This paper proposes an improved Recurrent Neural Network Transducer (RNN-T) model that integrates language bias directly into the end-to-end framework. By treating language identity as a predictable token, the model learns to handle code-switching transitions without requiring a complex, standalone LID module.

Proposed Methodology

The authors introduce three key modifications to the standard RNN-T architecture to bake in language awareness:

  1. Transcription Tagging: Language markers (e.g., <chn> and <eng>) are inserted into training transcriptions at every switch point. The model is trained to predict these tags just like subwords, turning language identification into a standard text prediction task.
  2. Word Embedding Constraints: To help the model distinguish between languages, a unique language vector is concatenated to the word embeddings. This mathematical "hint" forces the model to separate English subwords from Mandarin characters in its internal representation.
  3. Posterior Biasing during Inference: During decoding, the model uses its own predicted language IDs to "nudge" the probability of the next word. If the model predicts an <eng> tag, it increases the likelihood that subsequent words will be English.
Architecture comparison between standard RNN-T (a) and the proposed model with language bias (b).

Experimental Setup

The approach was evaluated on the SEAME (South East Asia Mandarin English) corpus, a benchmark for code-switching research featuring spontaneous conversational speech from Singapore and Malaysia.

  • Training Data: Approximately 101 hours of speech, tripled via speed perturbation (0.9x, 1.0x, 1.1x) to combat data scarcity.
  • Evaluation Sets: test_man (Mandarin-heavy) and test_sge (English-heavy with Southeast Asian accents).

Results and Analysis

The proposed "RNN-T with Language Bias" significantly outperformed the baseline models. By integrating LID directly into the prediction network, the model effectively functions as its own language identifier.

Mixed Error Rate (MER%) comparison across different models and data augmentation settings.

As shown in the results, the proposed method (RNN-T with LID ++Emb constraint) consistently reduced the Mixed Error Rate (MER) compared to the standard RNN-T (No-LID). With the addition of language model (LM) re-scoring, the system achieved a MER of 27.9% on the test_man set and 39.1% on the test_sge set. This represents a 16.2% and 12.9% relative error reduction, respectively, compared to the RNN-T baseline.

The authors noted that while the model handles code-switching transitions accurately, the performance gap on the test_sge set highlights the ongoing challenge of regional accents in end-to-end models.

Conclusion

This work demonstrates that language bias can be effectively integrated into an E2E RNN-T model, streamlining code-switching speech recognition. By eliminating the need for a separate LID module, the system achieves state-of-the-art performance with reduced complexity. Future research will focus on addressing accent-related performance gaps and exploring pre-training strategies to further improve robustness.

Summary updated

Abstract

Recently, language identity information has been utilized to improve the performance of end-to-end code-switching (CS) speech recognition. However, previous works use an additional language identification (LID) model as an auxiliary module, which causes the system complex. In this work, we propose an improved recurrent neural network transducer (RNN-T) model with language bias to alleviate the problem. We use the language identities to bias the model to predict the CS points. This promotes the model to learn the language identity information directly from transcription, and no additional LID model is needed. We evaluate the approach on a Mandarin-English CS corpus SEAME. Compared to our RNN-T baseline, the proposed method can achieve 16.2% and 12.9% relative error reduction on two test sets, respectively.
BibTeX
@inproceedings{zhang2021rnn,
  author = {Shuai Zhang and Jiangyan Yi and Zhengkun Tian and Jianhua Tao and Ye Bai},
  title = {{Rnn-transducer with language bias for end-to-end Mandarin-English code-switching speech recognition}},
  booktitle = {ISCSLP},
  year = {2021},
  url = {https://openreview.net/forum?id=9IGPIxs99W},
}
Figure