Language Modeling for Code-Switching: Evaluation, Integration of Monolingual Data, and Discriminative Training

Hila Gonen, Yoav Goldberg

Introduction

Code-switching (CS)—the practice of alternating between two or more languages in a single conversation—is a common linguistic phenomenon among bilingual speakers. Despite its prevalence, CS poses significant challenges for Automatic Speech Recognition (ASR) systems. Most current ASR models are trained on monolingual data and struggle at "switch points," often misidentifying words or failing to maintain grammatical coherence when languages mix.

This paper by Hila Gonen and Yoav Goldberg addresses the limitations of traditional language modeling in the context of code-switching. The authors identify three primary hurdles: the scarcity of large-scale CS datasets, the inadequacy of standard evaluation metrics like perplexity, and the limitations of generative modeling for ranking ASR hypotheses.

Rethinking Evaluation: The SpanglishEval Benchmark

Standard language models (LMs) are typically evaluated using perplexity, which measures how well a model predicts a sample. However, perplexity is highly sensitive to vocabulary size and does not necessarily correlate with an LM's ability to assist an ASR system in selecting the correct sentence from a list of candidates.

To address this, the authors propose a ranking-based evaluation. They introduced SpanglishEval, a benchmark derived from the Bangor Miami Corpus. For every "gold" code-switched sentence, the authors generated phonetically similar but linguistically incorrect "alternatives" using Finite-State Transducers (FSTs). The model's task is to correctly rank the gold sentence above these distractors.

Examples of gold code-switched sentences and their phonetically similar alternatives used in the SpanglishEval benchmark.

Methodology: Discriminative Training and Fine-Tuning

The core innovation of this work is the shift from generative to discriminative training. Instead of training the model to predict the next word (a generative task), the authors train it to maximize the score of the gold sentence relative to negative samples.

Using a BiLSTM architecture to represent sentences, they employ a hinge loss function that incorporates Word Error Rate (WER) as a margin. This approach penalizes the model more heavily if it assigns a high score to a sentence that is significantly different from the ground truth.

Furthermore, the researchers developed an effective integration strategy for monolingual data. They pre-train the model on large English and Spanish datasets (OpenSubtitles2018) with a shared vocabulary, followed by a fine-tuning phase on the limited available code-switched data. This sequence allows the model to leverage broad linguistic knowledge before specializing in the nuances of language switching.

Key Results

The experimental results demonstrate the superiority of the discriminative approach over standard generative models. The FINE-TUNED-DISCRIMINATIVE model achieved a test set accuracy of 75.5%, a substantial improvement over the 65.4% achieved by the best generative baseline (FINE-TUNED-LM).

Comparison of language model performance across perplexity, accuracy, and WER metrics on dev and test sets.

The improvement was particularly pronounced on code-switched sentences, where the discriminative model outperformed the generative baseline by over 27 percentage points. Additionally, the results confirmed that while perplexity is a poor indicator of ranking performance, the discriminative model effectively utilizes monolingual pre-training to gain a 5-point accuracy boost.

Conclusion and Future Directions

By treating language modeling as a ranking task rather than a next-word prediction task, Gonen and Goldberg show that models can be more efficient and accurate for ASR integration. While the current study uses synthetic negative samples, future work aims to test these models within live, end-to-end ASR systems and explore sub-word or character-level modeling to better handle out-of-vocabulary words common in spontaneous speech.

Summary updated

Abstract

We focus on the problem of language modeling for code-switched language, in the context of automatic speech recognition (ASR). Language modeling for code-switched language is challenging for (at least) three reasons: (1) lack of available large-scale code-switched data for training; (2) lack of a replicable evaluation setup that is ASR directed yet isolates language modeling performance from the other intricacies of the ASR system; and (3) the reliance on generative modeling. We tackle these three issues: we propose an ASR-motivated evaluation setup which is decoupled from an ASR system and the choice of vocabulary, and provide an evaluation dataset for English-Spanish code-switching. This setup lends itself to a discriminative training approach, which we demonstrate to work better than generative language modeling. Finally, we explore a variety of training protocols and verify the effectiveness of training with large amounts of monolingual data followed by fine-tuning with small amounts of code-switched data, for both the generative and discriminative cases.
BibTeX
@misc{gonen2018language,
  author = {Hila Gonen and Yoav Goldberg},
  title = {{Language Modeling for Code-Switching: Evaluation, Integration of Monolingual Data, and Discriminative Training}},
  year = {2018},
  eprint = {1810.11895},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.11895},
}
Figure