Translation Bias in Code-Switched Speech Recognition

This direction is merged into another direction: intra-sentential code-switching. Its last version stays readable and citable here.

Version 1 of 113 papers24 conceptsfrozen 2026-10-04

In many parts of the world, speakers naturally alternate between two or more languages in a single conversation—a phenomenon known as code-switching. While humans navigate these transitions effortlessly, modern Automatic Speech Recognition (ASR) systems often stumble.

Recent research has identified a persistent failure mode in large-scale multilingual models: instead of transcribing the specific words spoken in each language, the models often "helpfully" translate the entire utterance into a single dominant language. This translation bias represents a fundamental conflict between a model's ability to understand cross-lingual semantics and its ability to provide a faithful, verbatim transcript.

The Mechanism of Translation Bias

Large-scale models like Whisper are trained on massive datasets that include both transcription (speech-to-text) and translation (speech-to-translated-text) tasks. While this multi-task training makes the models powerful, it creates an internal tug-of-war.

As Hexin Liu and Haizhou Li observe in Code-switching Speech Recognition Under the Lens, these models develop linguistic priors that favor a "dominant" language. When the model encounters a code-switched segment—for instance, a Mandarin speaker inserting English technical terms—the model’s internal language model may decide that the English words are "out of place." To maintain monolingual consistency, the model erroneously translates the English terms back into Mandarin or translates the entire sentence into English.

This isn't just a minor error; it is a catastrophic failure of intent. For communities where language mixing is the standard colloquial form, a model that translates everything into English erases the linguistic reality of the speaker. Bashar Talafha and colleagues, in their work on the Casablanca dataset, found that even state-of-the-art models like Whisper-v3 struggle with this, often defaulting to English when faced with diverse Arabic dialects and code-switching.

The Conflict of Objectives

The problem is compounded by the "functional tension" between different AI tasks. In Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models, Guangzhi Sun and his team point out that training a model to follow spoken instructions can actually degrade its transcription accuracy. The model might start trying to execute a command it hears rather than simply transcribing it.

Similarly, Jiaming Zhou and Hui Wang argue in their work on the CS-Dialogue dataset that multi-task models often lose the ability to distinguish between the "transcribe" and "translate" prompts when processing bilingual input. The model's training objective might favor the most "likely" sequence of words in a single language over the acoustically accurate but linguistically "messy" reality of code-switching.

Mitigating the Bias

To move past this bias, researchers are pursuing three main strategies:

1. Targeted Data Augmentation Since real-world code-switched data is scarce, researchers use Text-to-Speech (TTS) to generate synthetic examples. Hexin Liu introduced SECT, a prompting technique that uses linguistic theory to generate realistic code-switched text for training. Meanwhile, Yupei Li and Zifan Wei developed the DOTA-ME-CS dataset, which uses AI to modify human recordings (changing speed and timbre) to create more robust benchmarks for Mandarin-English switching.

2. Specialized Loss Functions Rather than relying on standard cross-entropy loss, some researchers introduce explicit rewards for correctly identifying minority languages. Yash Sharma and Preethi Jyothi proposed a CS-bias loss function in Improving Low Resource Code-Switched ASR Using Augmented Code-Switched TTS. This mechanism encourages the model to recognize language transitions rather than defaulting to the phonology of the "matrix" (dominant) language.

3. Domain-Specific Fine-Tuning In high-stakes environments like aircraft cockpits, translation bias can be dangerous. Kartheek Kumar Reddy Nareddy and Julia Niebling demonstrated that applying Low-Rank Adaptation (LoRA) to Whisper models can significantly reduce this bias. By fine-tuning on technical, multilingual pilot speech, they forced the model to prioritize verbatim transcription over its internal urge to translate everything into English, drastically reducing error rates.

Why It Works: Moving Toward Acoustic Fidelity

The shift in this research area is away from "semantic understanding" (which leads to translation) and back toward "acoustic fidelity." By using language-specific backbones and guiding models with "unit language"—text-like representations of speech units as explored by Yuhao Zhang in Leveraging Unit Language Guidance—researchers are teaching models that the goal of transcription is to mirror the audio signal, not to "fix" the speaker's language choice.

Ultimately, solving translation bias is about ensuring that ASR systems respect the linguistic diversity of their users, transcribing the world as it is spoken, not as a monolingual model thinks it should be spoken.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_translation_bias_in_code_switched_speech_abff202a,
  title = {Translation Bias in Code-Switched Speech Recognition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/translation-bias-in-code-switched-speech-recognition/rdr_abff202a11754e0b810932a1d8e52197},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_translation_bias_in_code_switched_speech_abff202a_v1,
  title = {Translation Bias in Code-Switched Speech Recognition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/translation-bias-in-code-switched-speech-recognition/rdr_abff202a11754e0b810932a1d8e52197/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}