Introduction
Code-switching (CS)—the practice of alternating between two or more languages in a single conversation—is a common linguistic phenomenon among bilingual speakers. Despite its prevalence, CS poses significant challenges for Automatic Speech Recognition (ASR) systems. Most current ASR models are trained on monolingual data and struggle at "switch points," often misidentifying words or failing to maintain grammatical coherence when languages mix.
This paper by Hila Gonen and Yoav Goldberg addresses the limitations of traditional language modeling in the context of code-switching. The authors identify three primary hurdles: the scarcity of large-scale CS datasets, the inadequacy of standard evaluation metrics like perplexity, and the limitations of generative modeling for ranking ASR hypotheses.
Rethinking Evaluation: The SpanglishEval Benchmark
Standard language models (LMs) are typically evaluated using perplexity, which measures how well a model predicts a sample. However, perplexity is highly sensitive to vocabulary size and does not necessarily correlate with an LM's ability to assist an ASR system in selecting the correct sentence from a list of candidates.
To address this, the authors propose a ranking-based evaluation. They introduced SpanglishEval, a benchmark derived from the Bangor Miami Corpus. For every "gold" code-switched sentence, the authors generated phonetically similar but linguistically incorrect "alternatives" using Finite-State Transducers (FSTs). The model's task is to correctly rank the gold sentence above these distractors.
Methodology: Discriminative Training and Fine-Tuning
The core innovation of this work is the shift from generative to discriminative training. Instead of training the model to predict the next word (a generative task), the authors train it to maximize the score of the gold sentence relative to negative samples.
Using a BiLSTM architecture to represent sentences, they employ a hinge loss function that incorporates Word Error Rate (WER) as a margin. This approach penalizes the model more heavily if it assigns a high score to a sentence that is significantly different from the ground truth.
Furthermore, the researchers developed an effective integration strategy for monolingual data. They pre-train the model on large English and Spanish datasets (OpenSubtitles2018) with a shared vocabulary, followed by a fine-tuning phase on the limited available code-switched data. This sequence allows the model to leverage broad linguistic knowledge before specializing in the nuances of language switching.
Key Results
The experimental results demonstrate the superiority of the discriminative approach over standard generative models. The FINE-TUNED-DISCRIMINATIVE model achieved a test set accuracy of 75.5%, a substantial improvement over the 65.4% achieved by the best generative baseline (FINE-TUNED-LM).
The improvement was particularly pronounced on code-switched sentences, where the discriminative model outperformed the generative baseline by over 27 percentage points. Additionally, the results confirmed that while perplexity is a poor indicator of ranking performance, the discriminative model effectively utilizes monolingual pre-training to gain a 5-point accuracy boost.
Conclusion and Future Directions
By treating language modeling as a ranking task rather than a next-word prediction task, Gonen and Goldberg show that models can be more efficient and accurate for ASR integration. While the current study uses synthetic negative samples, future work aims to test these models within live, end-to-end ASR systems and explore sub-word or character-level modeling to better handle out-of-vocabulary words common in spontaneous speech.
Summary updated