When bilingual speakers alternate between languages mid-sentence—a phenomenon known as code-switching—they create a unique challenge for speech technologies. While massive monolingual datasets exist for high-resource languages, natural, spontaneous code-switched speech is notoriously scarce.
Standard speech recognition (ASR) and text-to-speech (TTS) models fail when confronted with code-switching because they are trained on uniform linguistic patterns. As Pascale Fung et al. observe in Language Modeling with Functional Head Constraint for Code Switching Speech Recognition, the specific points where speakers transition between languages are rare in training text. Furthermore, Saurabh Garg et al. point out in Dual Language Models for Code Switched Speech Recognition that higher-order context windows (like n-grams) fail because the sheer variability of language transitions means specific multi-language word sequences are rarely observed. When a switch occurs, the model's confidence plummets.
Early efforts focused on gathering ecological data, resulting in foundational corpora such as SEAME for Mandarin-English (Dau-Cheng Lyu et al. in SEAME: a Mandarin-English code-switching speech corpus in south-east asia), ArzEn for Egyptian Arabic-English (Injy Hamed et al. in ArzEn: A Speech Corpus for Code-switched Egyptian Arabic-English), and DECM for German-English (Enes Yavuz Ugan et al. in DECM: Evaluating Bilingual ASR Performance on a Code-switching/mixing Benchmark). However, collecting and manually annotating spontaneous speech across diverse language pairs is prohibitively expensive and cannot scale to match the volume of monolingual data.
To bypass this data bottleneck, researchers have developed techniques that leverage abundant monolingual resources through audio synthesis, modular architectures, script unification, and semi-supervised filtering.
Synthesizing Transitions: Audio Splicing and Collaging
If real-world code-switched audio is rare, can we manufacture it? A prominent strategy involves stitching together fragments of abundant monolingual speech to simulate code-switching.
The intuition is straightforward: if you have a recording of an English speaker saying "I want to buy some" and a Mandarin speaker saying "蘋果" (apples), you can concatenate the audio clips. Karan Taneja et al. demonstrated in Exploiting Monolingual Speech Corpora for Code-Mixed Speech Recognition that intelligently collating monolingual fragments to mimic natural linguistic transitions yields significant improvements in Word Error Rate (WER).
Monolingual English: [I want to buy some] [books]
Monolingual Mandarin: [我想要買] [蘋果]
⬇️ (Splicing / Collaging)
Synthetic Code-Switched: [I want to buy some] [蘋果]
Expanding on this, Amir Hussein et al. introduced "Speech Collage" (Speech Collage: Code-Switched Audio Generation by Collaging Monolingual Corpora), a framework that automates this splicing process to train models for zero-shot scenarios. This approach proves that exposing a model to artificial acoustic boundaries successfully teaches it to navigate real language shifts.
Architectural Separation: Mixture of Experts and Multi-Encoders
Rather than forcing a single neural network to master two entirely different phonetic spaces simultaneously with limited data, another approach splits the workload. Monolingual speech features are highly distinct (e.g., Mandarin tones versus English intonation). Forcing a single encoder to map both into the same latent space with limited data causes representational interference. By utilizing pre-trained monolingual models as "experts," architectures can handle each language independently and fuse them dynamically.
Xinyuan Zhou et al. pioneered this with a Multi-Encoder-Decoder Transformer (Multi-Encoder-Decoder Transformer for Code-Switching Speech Recognition), pre-training separate encoders on massive monolingual datasets. To optimize how the model switches between these encoders, Hao Li introduced a frame-level gating network using a Mixture of Experts (MoE) approach (Bi-Encoder Transformer Network for Mandarin-English Code-Switching Speech Recognition Using Mixture of Experts).
[Audio Input]
│
┌───────────────┴───────────────┐
▼ ▼
[Mandarin Encoder] [English Encoder]
(Pre-trained) (Pre-trained)
│ │
└───────────────┬───────────────┘
▼
[Gating Network / MoE]
│
▼
[Merged Output]
To address the computational latency of running multiple full encoders, Fengyun Tan et al. developed a token-level Lightweight Switch-Routing MoE (Improving End-to-End Modeling For Mandarin-English Code-Switching Using Lightweight Switch-Routing Mixture-of-Experts), which routes representations to language-specific specialists on the fly. Similarly, He Wang 0022 et al. designed CAMEL (CAMEL: Cross-Attention Enhanced Mixture-of-Experts and Language Bias for Code-Switching Speech Recognition), which uses gated cross-attention and a Language Diarization decoder to inject a "language bias," helping the model anticipate switches before they happen.
Bridging the Script Gap via Transliteration
Code-switching often involves languages with entirely different writing systems (e.g., Hindi in Devanagari and English in Latin script). This creates a severe lexical mismatch for end-to-end models, as the vocabulary space is split down the middle.
To bypass this, researchers use transliteration to map both languages into a shared textual space. If the model outputs everything in a single script, it does not have to learn two separate character sets and can share subword embeddings across languages.
For instance, in Text-to-Speech (TTS) synthesis, Raviraj Joshi et al. showed in Code-Mixed Text-to-Speech Synthesis Under Low-Resource Constraints that transliterating English text into Devanagari script allows a model to learn shared pronunciation patterns with minimal bilingual data. In speech recognition, Brian Yan et al. proposed "indiscriminate transliteration" (Towards Zero-Shot Code-Switched Speech Recognition), eliminating the need for explicit language segmentation by mapping acoustic features directly to a unified representation.
Filtering the Noise: Semi-Supervised Learning
When human-labeled data is exhausted, large-scale unlabeled data or pseudo-labeling becomes necessary. However, pseudo-labels generated by standard models in code-switched contexts are highly prone to "hallucinations" or language confusion at the transition boundaries.
To solve this, recent frameworks use intelligent filters to clean unsupervised data. Yu Xi et al. integrated a Large Language Model (LLM) as a data filter within a Noisy Student Training pipeline (Semi-supervised Learning for Code-Switching ASR with Large Language Model Filter), leveraging the LLM's internal grammatical knowledge to validate intra-sentential transitions.
Taking this a step further, Liang-Hsuan Tseng et al. developed the K2D framework (Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Data), which uses a small auxiliary model to double-check pseudo-labels across tens of thousands of hours of realistic, unlabeled audio. This successfully distills robust code-switching capabilities into highly efficient models without requiring clean, human-annotated data.
Key Foundational Work
Using English Phoneme Models for Chinese Speech Recognition (1998) — contributed 1 concept elements to this direction.
SEAME: a Mandarin-English code-switching speech corpus in south-east asia (2010) — contributed 1 concept elements to this direction.