Introduction
Code-switching (CS)—the fluid alternation between two languages within a single conversation—is a common linguistic phenomenon in bilingual communities. Despite its prevalence, CS presents significant challenges for Automatic Speech Recognition (ASR) systems. These systems rely on Language Models (LMs) to resolve acoustic ambiguities, yet standard LMs struggle with the unique syntactic and data-scarcity issues inherent in code-switched speech. This paper addresses these hurdles by proposing a novel evaluation framework, a discriminative training approach, and a strategy for leveraging monolingual data.
The Evaluation Challenge
Building LMs for code-switching is difficult for three primary reasons:
- Data Scarcity: CS is primarily a spoken phenomenon, making large-scale text datasets rare.
- Flawed Metrics: Standard metrics like perplexity are sensitive to vocabulary size, making it difficult to compare models across different language configurations. Furthermore, perplexity does not directly measure a model's ability to distinguish between a correct transcription and a phonetically similar error.
- Generative Mismatch: Standard LMs are trained to predict the next word (generative), whereas ASR tasks often require ranking a list of complete sentence hypotheses.
To address this, the authors introduce a ranking-based evaluation scheme. Instead of perplexity, the LM is tasked with ranking a "gold" (correct) sentence higher than a set of phonetically similar alternatives that simulate common ASR errors.
Methodology
Discriminative Training
Moving away from generative next-word prediction, the authors propose training a BiLSTM discriminatively to score entire sentences. Using a margin-based loss function, the model is penalized if the score of a negative (incorrect) example is too close to the gold sentence's score. This margin is scaled by the Word Error Rate (WER) of the negative example relative to the gold sentence, forcing the model to focus on the most plausible errors.
Monolingual Pre-training
To mitigate data scarcity, the authors employ a Pre-train then Fine-tune protocol. The model is first trained on massive monolingual corpora (English and Spanish separately but with shared parameters) to learn the underlying syntax and vocabulary of both languages. It is then fine-tuned on a smaller code-switched dataset to learn how the languages are mixed.
Experimental Setup and Datasets
The study utilizes the Bangor Miami Corpus (transcripts of bilingual English-Spanish speakers) for code-switched data and OpenSubtitles2018 for monolingual pre-training. Evaluation is conducted on SpanglishEval, a new benchmark containing 2,000 sets of gold sentences paired with synthetic, phonetically similar distractors in English, Spanish, and code-switched formats.
Results and Discussion
The experimental results demonstrate that discriminative training significantly outperforms generative baselines. As shown in the table below, the fine-tuned discriminative model achieved the highest accuracy and lowest WER across both development and test sets.
Key Findings
- Discriminative Superiority: The best generative LM reached 65.4% accuracy on the test set, while the discriminative model achieved 75.5%, a 10.1-point absolute improvement.
- The Power of Pre-training: Monolingual pre-training provided a crucial boost, particularly when code-switched training data was limited. For the discriminative model, it increased accuracy from 70.5% to 75.5%.
- Handling Language Mixing: The discriminative model showed a dramatic 58% relative improvement specifically on sentences that were code-switched, indicating it successfully learned the patterns of language alternation.
Conclusion and Limitations
This work demonstrates that ranking-based evaluation and discriminative training are more aligned with the needs of ASR systems for code-switched speech than traditional generative modeling. While the authors acknowledge that their synthetic negative examples might introduce specific artifacts that the model learns to exploit, the substantial gains over baselines suggest a robust improvement in linguistic modeling. Future research will focus on better handling out-of-vocabulary tokens and integrating this framework into end-to-end ASR pipelines.
Summary updated