Abstract
Code switching (CS) refers to the phenomenon of interchangeably using words and phrases from different languages. CS can pose significant accuracy challenges to NLP, due to the often monolingual nature of the underlying systems. In this work, we focus on CS in the context of English/Spanish conversations for the task of speech translation (ST), generating and evaluating both transcript and translation. To evaluate model performance on this task, we create a novel ST corpus derived from existing public data sets. We explore various ST architectures across two dimensions: cascaded (transcribe then translate) vs end-to-end (jointly transcribe and translate) and unidirectional (source -> target) vs bidirectional (source <-> target). We show that our ST architectures, and especially our bidirectional end-to-end architecture, perform well on CS speech, even when no CS training data is used.
Introduction
Over half the world’s population is bilingual, and for many, "code-switching"—the fluid transition between two or more languages within a single conversation—is a natural part of daily communication. Despite its prevalence, code-switching remains a significant hurdle for modern Speech Translation (ST) systems. Most existing models are built on a monolingual foundation, assuming a single "matrix" language. When these systems encounter mixed-language input, such as "Spanglish," they frequently fail to recognize foreign words or propagate errors through the translation pipeline.
This paper addresses these challenges by investigating robust architectures for English/Spanish code-switched speech translation. We specifically examine the benefits of end-to-end (E2E) modeling and bidirectional training to bridge the gap between recognition and translation.
Methodology
Architectural Dimensions
We evaluate two primary architectural paradigms:
- Cascaded vs. End-to-End (E2E): Traditional cascaded systems utilize an Automatic Speech Recognition (ASR) module to generate a transcript, which is then passed to a Machine Translation (MT) module. In contrast, our E2E systems employ a "triangle" architecture that jointly generates both the transcript and the translation directly from the audio signal.
- Unidirectional vs. Bidirectional: While unidirectional models are specialized for one language pair (e.g., Spanish to English), bidirectional models are trained to handle both directions ($Source \leftrightarrow Target$) simultaneously.
Model Components
Our models leverage powerful pre-trained modules to maximize performance. We use Wav2Vec 2.0 as the audio encoder and mBART 50-50 for the text decoder. Our most sophisticated configuration, the E2E Bidirectional Shared model, utilizes a single bilingual decoder for both transcription and translation tasks. This design encourages the model to learn an implicit Language Identification (LID) capability, avoiding the need for a separate, error-prone LID classifier.
Data and Evaluation
Due to the lack of dedicated code-switched ST datasets, we derived two novel evaluation sets from existing corpora:
- Fisher CS: Extracted from a Spanish ASR dataset, this set contains utterances with English code-switching. While the audio is "noisy" (phone-call quality), the transcriptions are relatively clean.
- Bangor Miami: A corpus of natural, casual conversations between bilingual speakers in Florida. This dataset represents "in-the-wild" speech with high proportions of natural code-switching and significant background noise.
Initial training was conducted on standard monolingual ST datasets, including CoVoST (Spanish $\rightarrow$ English) and MuST-C (English $\rightarrow$ Spanish), before testing on the code-switched subsets.
Key Insights and Results
Performance Gains
The E2E Bidirectional Shared model emerged as the top performer. On the Fisher CS test set, it achieved a 33.8 Word Error Rate (WER), significantly outperforming the best cascaded system, which recorded a 37.1 WER. When fine-tuned on a small amount of code-switched data, the E2E model's performance improved further to 30.0 WER and 25.4 BLEU.
Robustness to Code-Switching Density
A critical finding was the model's stability across varying degrees of code-switching. Accuracy did not degrade as the density of code-switched words increased; whether a sentence contained 5% or 45% mixed-language content, the model maintained consistent performance. This suggests that the shared bilingual parameters provide a robust inductive bias for handling linguistic fluidity.
Implicit vs. Explicit LID
Traditional systems often rely on an explicit Language Identification step to route input to the correct monolingual model. Our research indicates that letting the E2E model learn to identify languages implicitly during the decoding process results in fewer errors and better recovery from recognition failures. Cascaded models often "hallucinated" when the ASR step misidentified a code-switched word, whereas E2E models were more likely to provide an accurate translation despite transcript errors.
Conclusion and Future Work
Our findings demonstrate that end-to-end, bidirectional architectures are highly effective for code-switched speech translation, even in zero-shot scenarios where no specific CS training data is available. By sharing parameters across languages and tasks, these models develop the flexibility required to navigate the complexities of bilingual speech.
Future research will expand this work to language pairs with different scripts and explore "intra-word" switching (e.g., mixing morphology from two languages). Additionally, improving performance on low-quality, naturalistic audio remains a priority for making these systems viable for real-world applications.
Summary updated