Language Modeling for Code-Switching: Evaluation, Integration of Monolingual Data, and Discriminative Training

Hila Gonen, Yoav Goldberg

Introduction

Code-switching (CS)—the fluid alternation between two languages within a single conversation—is a common linguistic phenomenon in bilingual communities. Despite its prevalence, CS presents significant challenges for Automatic Speech Recognition (ASR) systems. These systems rely on Language Models (LMs) to resolve acoustic ambiguities, yet standard LMs struggle with the unique syntactic and data-scarcity issues inherent in code-switched speech. This paper addresses these hurdles by proposing a novel evaluation framework, a discriminative training approach, and a strategy for leveraging monolingual data.

The Evaluation Challenge

Building LMs for code-switching is difficult for three primary reasons:

  1. Data Scarcity: CS is primarily a spoken phenomenon, making large-scale text datasets rare.
  2. Flawed Metrics: Standard metrics like perplexity are sensitive to vocabulary size, making it difficult to compare models across different language configurations. Furthermore, perplexity does not directly measure a model's ability to distinguish between a correct transcription and a phonetically similar error.
  3. Generative Mismatch: Standard LMs are trained to predict the next word (generative), whereas ASR tasks often require ranking a list of complete sentence hypotheses.

To address this, the authors introduce a ranking-based evaluation scheme. Instead of perplexity, the LM is tasked with ranking a "gold" (correct) sentence higher than a set of phonetically similar alternatives that simulate common ASR errors.

Examples of gold sentences and phonetically similar alternatives from the SpanglishEval benchmark.

Methodology

Discriminative Training

Moving away from generative next-word prediction, the authors propose training a BiLSTM discriminatively to score entire sentences. Using a margin-based loss function, the model is penalized if the score of a negative (incorrect) example is too close to the gold sentence's score. This margin is scaled by the Word Error Rate (WER) of the negative example relative to the gold sentence, forcing the model to focus on the most plausible errors.

Monolingual Pre-training

To mitigate data scarcity, the authors employ a Pre-train then Fine-tune protocol. The model is first trained on massive monolingual corpora (English and Spanish separately but with shared parameters) to learn the underlying syntax and vocabulary of both languages. It is then fine-tuned on a smaller code-switched dataset to learn how the languages are mixed.

Experimental Setup and Datasets

The study utilizes the Bangor Miami Corpus (transcripts of bilingual English-Spanish speakers) for code-switched data and OpenSubtitles2018 for monolingual pre-training. Evaluation is conducted on SpanglishEval, a new benchmark containing 2,000 sets of gold sentences paired with synthetic, phonetically similar distractors in English, Spanish, and code-switched formats.

Results and Discussion

The experimental results demonstrate that discriminative training significantly outperforms generative baselines. As shown in the table below, the fine-tuned discriminative model achieved the highest accuracy and lowest WER across both development and test sets.

Main results comparing generative language models (LM) and discriminative models (disc) across perplexity, accuracy, and word error rate.

Key Findings

  • Discriminative Superiority: The best generative LM reached 65.4% accuracy on the test set, while the discriminative model achieved 75.5%, a 10.1-point absolute improvement.
  • The Power of Pre-training: Monolingual pre-training provided a crucial boost, particularly when code-switched training data was limited. For the discriminative model, it increased accuracy from 70.5% to 75.5%.
  • Handling Language Mixing: The discriminative model showed a dramatic 58% relative improvement specifically on sentences that were code-switched, indicating it successfully learned the patterns of language alternation.

Conclusion and Limitations

This work demonstrates that ranking-based evaluation and discriminative training are more aligned with the needs of ASR systems for code-switched speech than traditional generative modeling. While the authors acknowledge that their synthetic negative examples might introduce specific artifacts that the model learns to exploit, the substantial gains over baselines suggest a robust improvement in linguistic modeling. Future research will focus on better handling out-of-vocabulary tokens and integrating this framework into end-to-end ASR pipelines.

Summary updated

BibTeX
@misc{gonen2018language,
  author = {Hila Gonen and Yoav Goldberg},
  title = {{Language Modeling for Code-Switching: Evaluation, Integration of Monolingual Data, and Discriminative Training}},
  year = {2018},
  eprint = {1810.11895},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.11895},
}
Figure