End-to-End Speech Translation for Code Switched Speech

Orion Weller, Matthias Sperber, Telmo Pires, Hendra Setiawan, Christian Gollan, Dominic Telaar, Matthias Paulik

Abstract

Code-switching (CS), the fluid mixing of languages in conversation, poses a significant challenge for traditional monolingual speech translation systems. This paper explores end-to-end (E2E) architectures that leverage shared bidirectional parameters to bridge the language gap. By utilizing a "triangle" architecture with pre-trained foundations like Wav2Vec 2.0 and mBART, the proposed E2E Bidirectional Shared model outperforms cascaded systems on natural CS datasets like Fisher and Miami, demonstrating robustness to varying code-switching densities.

Introduction

Code-switching is a global reality; over half the world's population is bilingual, and many interchangeably use words from different languages in a single conversation (e.g., "Acá te tiene como constantemente escribiendo papers"). However, most Natural Language Processing (NLP) systems are "monolingual by design."

Standard Speech Translation (ST) usually involves a cascaded approach: first, an Automatic Speech Recognition (ASR) system transcribes the audio, and then a Machine Translation (MT) system translates that text. If the ASR system is trained only on Spanish, it will likely mangle any English words it hears. This "error propagation" means the subsequent translation is doomed from the start. Furthermore, there is a severe lack of high-quality, natural datasets for code-switched speech translation.

Methodology

The researchers explored several architectures to find the best way to handle linguistic mixing, focusing on increasing parameter sharing and moving toward end-to-end designs.

The Triangle Architecture

Instead of a linear pipeline, the authors use an E2E model that jointly generates both a transcript (in the original mixed languages) and a translation (into a single target language). This "triangle" setup allows the model to leverage information from both tasks simultaneously.

Pre-trained Foundations and Bidirectional Sharing

The models leverage powerful self-supervised foundations: Wav2Vec 2.0 for the audio encoder and mBART 50-50 for the text decoder. The core innovation is the E2E Bidirectional Shared model. Unlike "unidirectional" models that use separate components for each language, this model shares parameters across both languages and tasks, allowing it to learn a unified representation of the bilingual space.

Comparison of model architectures from cascaded to end-to-end with varying degrees of parameter sharing.

Datasets

The study introduces two "natural" CS datasets derived from existing public corpora, moving away from synthetic "Spanglish":

  • Fisher: Originally for ASR, repurposed for ST. It features Spanish speakers in the US/Canada with significant English and Spanish code-switching.
  • Bangor Miami: A corpus of casual conversations between bilinguals in Florida. Recorded in real-world settings with lapel mics, it provides a high-difficulty, "noisy" benchmark.

Key Results and Analysis

The E2E Bidirectional Shared model emerged as the clear winner across multiple metrics, proving that the more parameters the model shared between languages, the better it performed.

Performance Gains

On the Fisher dataset, the model achieved a 33.8 WER and 23.3 BLEU score without any fine-tuning on code-switched data. After fine-tuning, these improved to 30.0 WER and 25.4 BLEU. Despite the heavy noise in the Miami dataset, the model achieved an 11.8 BLEU score, significantly outperforming the 8.8 BLEU of a standard cascaded system.

Code-switching span accuracy for Cascade Bidirectional vs. E2E Bidirectional Shared models across different evaluation settings.

Robustness to CS Density

One of the most surprising findings was that the model's accuracy did not correlate with the proportion of code-switching in a sentence. Whether a sentence was 5% English or 45% English, the model handled it with similar proficiency.

The Advantage of E2E and Implicit LID

End-to-end models consistently beat cascaded models because they avoid "hard" errors in the middle of the process. The E2E decoder can use original audio features to "correct" its understanding of a mixed-language phrase even if the internal transcription is slightly off. Furthermore, models that learn to identify and switch between languages implicitly during decoding were found to be superior to those using explicit Language Identification (LID) "gatekeepers."

Conclusion and Future Work

While the E2E Bidirectional Shared model shows great promise, limitations remain. The study focused exclusively on Spanish-English; future work is needed to see if these findings hold for linguistically distant pairs like Hindi-English. Additionally, "intra-word" code-switching and low-resource constraints in "in-the-wild" audio continue to be challenges. Future research will also explore translating code-switched speech into target languages other than English.

Summary updated

Abstract

Code switching (CS) refers to the phenomenon of interchangeably using words and phrases from different languages. CS can pose significant accuracy challenges to NLP, due to the often monolingual nature of the underlying systems. In this work, we focus on CS in the context of English/Spanish conversations for the task of speech translation (ST), generating and evaluating both transcript and translation. To evaluate model performance on this task, we create a novel ST corpus derived from existing public data sets. We explore various ST architectures across two dimensions: cascaded (transcribe then translate) vs end-to-end (jointly transcribe and translate) and unidirectional (source -> target) vs bidirectional (source <-> target). We show that our ST architectures, and especially our bidirectional end-to-end architecture, perform well on CS speech, even when no CS training data is used.
BibTeX
@inproceedings{weller2022end,
  author = {Orion Weller and Matthias Sperber and Telmo Pires and Hendra Setiawan and Christian Gollan and Dominic Telaar and Matthias Paulik},
  title = {{End-to-End Speech Translation for Code Switched Speech}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=Khi-tSR928},
}
Figure