Text-Derived Language Identity Incorporation for End-to-End Code-Switching Speech Recognition

Qinyi Wang, Haizhou LiCALCS @ EMNLP 2023 · OpenReview

Abstract

Recognizing code-switching (CS) speech often presents challenges for an automatic speech recognition system (ASR) due to limited linguistic context in short monolingual segments, resulting in language confusion. To mitigate this issue, language identity (LID) is often integrated into the speech recognition system to provide additional linguistic context. However, previous works predominately focus on extracting language identity from speech signals. We introduce a novel approach to learn language identity from pure text data via a dedicated language identity-language model. Besides, we explore two strategies: LID state fusion and language posterior biasing, to integrate the text-derived language identities into the end-to-end ASR system. By incorporating hypothesized language identities, our ASR system gains crucial contextual cues, effectively capturing language transitions and patterns within code-switched utterances. We conduct speech recognition experiments on the SEAME corpus and demonstrate the effectiveness of our proposed methods. Our results reveal significantly improved transcriptions in code-switching scenarios, underscoring the potential of text-derived LID in enhancing code-switching speech recognition.

Introduction

Code-switching (CS)—the practice of alternating between two or more languages in a single conversation—is a significant hurdle for Automatic Speech Recognition (ASR) systems. When a speaker flips from Mandarin to English for just a few words, the "monolingual context" is often too short for the system to accurately identify the language, leading to "language confusion." This often results in the AI transcribing a word from one language as a phonetically similar word from another.

While researchers typically attempt to extract Language Identity (LID) from the audio signal, this paper argues that pure text data is an underutilized resource. Text contains the syntactic patterns of how people switch languages, providing the linguistic cues an ASR system needs to maintain accuracy during transitions.

Methodology: The LID-LM and State Fusion

The researchers proposed a two-part solution to integrate text-derived linguistic knowledge into a standard ASR framework.

1. The Language Identity-Language Model (LID-LM)

Instead of a standard language model that only predicts the next word, the authors developed the LID-LM. This model is trained to predict an alternating sequence of language tags and words: [Language ID] -> [Word] -> [Language ID] -> [Word]. This structure forces the model to learn the probability of a language switch occurring at any given point in a sentence.

Comparison between a standard Language Model (LM) and the proposed Language Identity-Language Model (LID-LM).

2. Integration via LID State Fusion

To "plug" this text-based knowledge into an End-to-End (E2E) Transformer-based ASR system, the authors introduced LID State Fusion. This is a gating mechanism that combines the internal hidden states of the LID-LM and the ASR decoder.

The proposed ASR architecture featuring LID State Fusion between the Decoder and LID-LM blocks.

A sigmoid-based gate decides how much weight to give the language-identity information at each step. This dynamic approach allows the system to rely on LID cues when the acoustic signal is weak but ignore them if the audio is clear, avoiding the error propagation issues found in simpler biasing methods.

Experimental Evaluation

The study utilized the SEAME corpus, a 110-hour dataset of spontaneous Mandarin-English code-switching speech. Performance was measured using the Mix Error Rate (MER), which combines Word Error Rate for English and Character Error Rate for Chinese.

The results demonstrated that the LID State Fusion method significantly outperformed the baseline Transformer ASR:

  • Baseline MER: 21.4% (eval_man) / 29.5% (eval_sge)
  • LID State Fusion MER: 20.4% (eval_man) / 28.2% (eval_sge)

This represents a relative error reduction of approximately 4.4% to 4.7%. The system proved particularly adept at "rescuing" short segments where phonetic similarities caused the baseline to fail.

Conclusion

This research highlights the potential of using text-derived language identity to enhance code-switching speech recognition. By capturing the linguistic patterns of language transitions through an LID-LM and integrating them via a sophisticated gating mechanism, the proposed system achieves notable improvements in transcription accuracy without requiring additional audio training data. Future work may explore pre-training on larger text corpora to further improve the model's predictive power.

Summary updated

Abstract

Recognizing code-switching (CS) speech often presents challenges for an automatic speech recognition system (ASR) due to limited linguistic context in short monolingual segments, resulting in language confusion. To mitigate this issue, language identity (LID) is often integrated into the speech recognition system to provide additional linguistic context. However, previous works predominately focus on extracting language identity from speech signals. We introduce a novel approach to learn language identity from pure text data via a dedicated language identity-language model. Besides, we explore two strategies: LID state fusion and language posterior biasing, to integrate the text-derived language identities into the end-to-end ASR system. By incorporating hypothesized language identities, our ASR system gains crucial contextual cues, effectively capturing language transitions and patterns within code-switched utterances. We conduct speech recognition experiments on the SEAME corpus and demonstrate the effectiveness of our proposed methods. Our results reveal significantly improved transcriptions in code-switching scenarios, underscoring the potential of text-derived LID in enhancing code-switching speech recognition.
BibTeX
@inproceedings{wang2023text,
  author = {Qinyi Wang and Haizhou Li},
  title = {{Text-Derived Language Identity Incorporation for End-to-End Code-Switching Speech Recognition}},
  booktitle = {CALCS @ EMNLP},
  year = {2023},
  url = {https://openreview.net/forum?id=TqHwXep44H},
}
Figure