Traditional speech recognition models thrive on clean, monolingual datasets. However, a significant portion of the world's population is bilingual or multilingual, frequently engaging in code-switching—the practice of alternating between two or more languages within a single conversation or even a single sentence.
Training automatic speech recognition (ASR) and spoken dialogue models to understand code-switched speech requires specialized corpora. Building these datasets presents unique challenges: speakers change their behavior when recorded, real-world data is highly unbalanced, and manual partitioning can easily bias evaluation metrics. Modern research addresses these bottlenecks through structured elicitation, in-the-wild harvesting, and algorithmic dataset balancing.
Elicitation Strategies and the Observer's Paradox
Gathering code-switched speech in a laboratory setting is notoriously difficult due to the observer's paradox: speakers often revert to formal, monolingual registers when they know they are being evaluated. To overcome this, researchers design structured environments that lower a speaker's cognitive guard and trigger natural language alternation.
One approach relies on multi-phase conversational engineering. In ZAEBUC-Spoken: A Multilingual Multidialectal Arabic-English Speech Corpus, the authors utilized a multi-phase meeting simulation. By transitioning bilingual pairs from a formal task introduction to an informal, peer-to-peer brainstorming session, the framework successfully captured both formal registers and highly spontaneous, intra-sentential code-switching within a single session. Similarly, the authors of Addressing Code-Switching in French/Algerian Arabic Speech established a comfortable sociolinguistic context by using targeted questions delivered by a bilingual linguist, encouraging natural language alternation rather than rigid translations.
Another powerful strategy leverages linguistic accommodation—the human tendency to subconsciously mirror the speaking style, vocabulary, and language-mixing patterns of a conversational partner. By deploying interactive dialogue systems that actively code-switch, researchers can predictably elicit mixed-language responses from users.
[Interactive Agent] -> "Jee haan, what would you like to order?" (Mixes Hindi/English)
│
▼ (Induces Linguistic Accommodation)
[Human User] -> "I'd like a coffee, aur ek muffin bhi por favor."
As demonstrated in Jee haan, I’d like both, por favor: Elicitation of a Code-Switched Corpus of Hindi–English and Spanish–English Human–Machine Dialog, crowdsourcing platforms can scale this interaction by embedding bilingual agents within simulated real-world scenarios like ordering food. The authors of Understanding Linguistic Accommodation in Code-Switched Human-Machine Dialogues formalized this phenomenon, showing that users systematically adapt their code-switching frequency and style to match the machine agent, providing a reliable pipeline for generating targeted conversational speech data.
Harvesting Spontaneous Speech In the Wild
When laboratory elicitation is insufficient, public media and digital platforms offer a rich repository of unscripted, multilingual interactions. Digital video platforms bridge the gap between formal written text and highly spontaneous speech, providing a robust representation of modern linguistic environments (Nohejl et al.).
Consequently, researchers have increasingly turned to broadcast media, podcasts, and online videos to construct benchmarks. For example, the authors of KSC2: An Industrial-Scale Open-Source Kazakh Speech Corpus integrated television news, radio programs, and podcasts to build a large-scale corpus capturing authentic Kazakh-Russian code-switching.
For evaluating ASR performance under realistic conditions, Ugan et al. curated the DECM benchmark from spontaneous YouTube videos, categorizing utterances into low, mid, and high code-switching levels based on the proportion of embedded English words. In highly multilingual societies, even entertainment media can serve as a primary source: Jansen van Vüren et al. utilized a 50-hour corpus derived from South African soap operas, capturing complex intra-sentential switches across eight distinct languages.
Algorithmic Partitioning and Balancing
Once a corpus is collected, preparing it for machine learning introduces unique structural challenges. In standard monolingual datasets, partitioning data into training, development, and test sets is straightforward. In code-switched corpora, however, a naive random split can easily unbalance the representation of specific language pairs, speaker demographics, or switching frequencies, leading to high variance and unreliable evaluation metrics.
To solve this, Jansen van Vüren et al. formulated dataset partitioning as a mixed-integer linear programming (MIP) problem. Instead of relying on random assignment, the MIP framework treats dataset splitting as a constrained optimization problem, mathematically guaranteeing that the training, development, and test sets maintain identical proportions of code-switched speech patterns and speaker diversity.
Furthermore, evaluating these models requires careful dataset structuring; Sailor et al. demonstrated the utility of incorporating "blind" evaluation sets that contain mixed utterances without explicit language labels, forcing the underlying ASR system to handle language identification and transcription simultaneously.
Scaling Through Synthetic Generation
Despite creative elicitation and harvesting techniques, human-annotated code-switched data remains scarce, particularly for low-resource language pairs. To bridge this gap, researchers are exploring algorithmic generation to scale up training corpora synthetically.
The authors of UniCoM: A Universal Code-Switching Speech Generator developed a pipeline powered by a text-mixing algorithm called SWORDS. By synthetically combining monolingual speech samples at grammatically viable insertion points, the framework generated CS-FLEURS, a massively multilingual code-switching corpus spanning 253 language pairs. This synthetic approach provides a scalable path forward, allowing speech technologies to generalize to diverse language pairs without requiring thousands of hours of manual recording.
Key Foundational Work
Language, gender, speaking style and language proficiency as factors influencing the autonomous vocalic filler production in spontaneous speech (2006) — contributed 1 concept elements to this direction.
Mandarin/English mixed-lingual name recognition for mobile phone (2005) — contributed 1 concept elements to this direction.