Abstract
Code-switching (CS), the fluid mixing of languages in conversation, poses a significant challenge for traditional monolingual speech translation systems. This paper explores end-to-end (E2E) architectures that leverage shared bidirectional parameters to bridge the language gap. By utilizing a "triangle" architecture with pre-trained foundations like Wav2Vec 2.0 and mBART, the proposed E2E Bidirectional Shared model outperforms cascaded systems on natural CS datasets like Fisher and Miami, demonstrating robustness to varying code-switching densities.
Introduction
Code-switching is a global reality; over half the world's population is bilingual, and many interchangeably use words from different languages in a single conversation (e.g., "Acá te tiene como constantemente escribiendo papers"). However, most Natural Language Processing (NLP) systems are "monolingual by design."
Standard Speech Translation (ST) usually involves a cascaded approach: first, an Automatic Speech Recognition (ASR) system transcribes the audio, and then a Machine Translation (MT) system translates that text. If the ASR system is trained only on Spanish, it will likely mangle any English words it hears. This "error propagation" means the subsequent translation is doomed from the start. Furthermore, there is a severe lack of high-quality, natural datasets for code-switched speech translation.
Methodology
The researchers explored several architectures to find the best way to handle linguistic mixing, focusing on increasing parameter sharing and moving toward end-to-end designs.
The Triangle Architecture
Instead of a linear pipeline, the authors use an E2E model that jointly generates both a transcript (in the original mixed languages) and a translation (into a single target language). This "triangle" setup allows the model to leverage information from both tasks simultaneously.
Pre-trained Foundations and Bidirectional Sharing
The models leverage powerful self-supervised foundations: Wav2Vec 2.0 for the audio encoder and mBART 50-50 for the text decoder. The core innovation is the E2E Bidirectional Shared model. Unlike "unidirectional" models that use separate components for each language, this model shares parameters across both languages and tasks, allowing it to learn a unified representation of the bilingual space.
Datasets
The study introduces two "natural" CS datasets derived from existing public corpora, moving away from synthetic "Spanglish":
- Fisher: Originally for ASR, repurposed for ST. It features Spanish speakers in the US/Canada with significant English and Spanish code-switching.
- Bangor Miami: A corpus of casual conversations between bilinguals in Florida. Recorded in real-world settings with lapel mics, it provides a high-difficulty, "noisy" benchmark.
Key Results and Analysis
The E2E Bidirectional Shared model emerged as the clear winner across multiple metrics, proving that the more parameters the model shared between languages, the better it performed.
Performance Gains
On the Fisher dataset, the model achieved a 33.8 WER and 23.3 BLEU score without any fine-tuning on code-switched data. After fine-tuning, these improved to 30.0 WER and 25.4 BLEU. Despite the heavy noise in the Miami dataset, the model achieved an 11.8 BLEU score, significantly outperforming the 8.8 BLEU of a standard cascaded system.
Robustness to CS Density
One of the most surprising findings was that the model's accuracy did not correlate with the proportion of code-switching in a sentence. Whether a sentence was 5% English or 45% English, the model handled it with similar proficiency.
The Advantage of E2E and Implicit LID
End-to-end models consistently beat cascaded models because they avoid "hard" errors in the middle of the process. The E2E decoder can use original audio features to "correct" its understanding of a mixed-language phrase even if the internal transcription is slightly off. Furthermore, models that learn to identify and switch between languages implicitly during decoding were found to be superior to those using explicit Language Identification (LID) "gatekeepers."
Conclusion and Future Work
While the E2E Bidirectional Shared model shows great promise, limitations remain. The study focused exclusively on Spanish-English; future work is needed to see if these findings hold for linguistically distant pairs like Hindi-English. Additionally, "intra-word" code-switching and low-resource constraints in "in-the-wild" audio continue to be challenges. Future research will also explore translating code-switched speech into target languages other than English.
Summary updated