Building Corpora for Code-Switched Speech Recognition

37 concepts34 papers2 proposals

Traditional speech recognition models thrive on clean, monolingual datasets. However, a significant portion of the world's population is bilingual or multilingual, frequently engaging in code-switching—the practice of alternating between two or more languages within a single conversation or even a single sentence.

Training automatic speech recognition (ASR) and spoken dialogue models to understand code-switched speech requires specialized corpora. Building these datasets presents unique challenges: speakers change their behavior when recorded, real-world data is highly unbalanced, and manual partitioning can easily bias evaluation metrics. Modern research addresses these bottlenecks through structured elicitation, in-the-wild harvesting, and algorithmic dataset balancing.

Elicitation Strategies and the Observer's Paradox

Gathering code-switched speech in a laboratory setting is notoriously difficult due to the observer's paradox: speakers often revert to formal, monolingual registers when they know they are being evaluated. To overcome this, researchers design structured environments that lower a speaker's cognitive guard and trigger natural language alternation.

One approach relies on multi-phase conversational engineering. In ZAEBUC-Spoken: A Multilingual Multidialectal Arabic-English Speech Corpus, the authors utilized a multi-phase meeting simulation. By transitioning bilingual pairs from a formal task introduction to an informal, peer-to-peer brainstorming session, the framework successfully captured both formal registers and highly spontaneous, intra-sentential code-switching within a single session. Similarly, the authors of Addressing Code-Switching in French/Algerian Arabic Speech established a comfortable sociolinguistic context by using targeted questions delivered by a bilingual linguist, encouraging natural language alternation rather than rigid translations.

Another powerful strategy leverages linguistic accommodation—the human tendency to subconsciously mirror the speaking style, vocabulary, and language-mixing patterns of a conversational partner. By deploying interactive dialogue systems that actively code-switch, researchers can predictably elicit mixed-language responses from users.

[Interactive Agent] -> "Jee haan, what would you like to order?" (Mixes Hindi/English)
                                    │
                                    ▼ (Induces Linguistic Accommodation)
[Human User]        -> "I'd like a coffee, aur ek muffin bhi por favor."

As demonstrated in Jee haan, I’d like both, por favor: Elicitation of a Code-Switched Corpus of Hindi–English and Spanish–English Human–Machine Dialog, crowdsourcing platforms can scale this interaction by embedding bilingual agents within simulated real-world scenarios like ordering food. The authors of Understanding Linguistic Accommodation in Code-Switched Human-Machine Dialogues formalized this phenomenon, showing that users systematically adapt their code-switching frequency and style to match the machine agent, providing a reliable pipeline for generating targeted conversational speech data.

Harvesting Spontaneous Speech In the Wild

When laboratory elicitation is insufficient, public media and digital platforms offer a rich repository of unscripted, multilingual interactions. Digital video platforms bridge the gap between formal written text and highly spontaneous speech, providing a robust representation of modern linguistic environments (Nohejl et al.).

Consequently, researchers have increasingly turned to broadcast media, podcasts, and online videos to construct benchmarks. For example, the authors of KSC2: An Industrial-Scale Open-Source Kazakh Speech Corpus integrated television news, radio programs, and podcasts to build a large-scale corpus capturing authentic Kazakh-Russian code-switching.

For evaluating ASR performance under realistic conditions, Ugan et al. curated the DECM benchmark from spontaneous YouTube videos, categorizing utterances into low, mid, and high code-switching levels based on the proportion of embedded English words. In highly multilingual societies, even entertainment media can serve as a primary source: Jansen van Vüren et al. utilized a 50-hour corpus derived from South African soap operas, capturing complex intra-sentential switches across eight distinct languages.

Algorithmic Partitioning and Balancing

Once a corpus is collected, preparing it for machine learning introduces unique structural challenges. In standard monolingual datasets, partitioning data into training, development, and test sets is straightforward. In code-switched corpora, however, a naive random split can easily unbalance the representation of specific language pairs, speaker demographics, or switching frequencies, leading to high variance and unreliable evaluation metrics.

To solve this, Jansen van Vüren et al. formulated dataset partitioning as a mixed-integer linear programming (MIP) problem. Instead of relying on random assignment, the MIP framework treats dataset splitting as a constrained optimization problem, mathematically guaranteeing that the training, development, and test sets maintain identical proportions of code-switched speech patterns and speaker diversity.

Furthermore, evaluating these models requires careful dataset structuring; Sailor et al. demonstrated the utility of incorporating "blind" evaluation sets that contain mixed utterances without explicit language labels, forcing the underlying ASR system to handle language identification and transcription simultaneously.

Scaling Through Synthetic Generation

Despite creative elicitation and harvesting techniques, human-annotated code-switched data remains scarce, particularly for low-resource language pairs. To bridge this gap, researchers are exploring algorithmic generation to scale up training corpora synthetically.

The authors of UniCoM: A Universal Code-Switching Speech Generator developed a pipeline powered by a text-mixing algorithm called SWORDS. By synthetically combining monolingual speech samples at grammatically viable insertion points, the framework generated CS-FLEURS, a massively multilingual code-switching corpus spanning 253 language pairs. This synthetic approach provides a scalable path forward, allowing speech technologies to generalize to diverse language pairs without requiring thousands of hours of manual recording.

Key Foundational Work

Language, gender, speaking style and language proficiency as factors influencing the autonomous vocalic filler production in spontaneous speech (2006) — contributed 1 concept elements to this direction.
Mandarin/English mixed-lingual name recognition for mobile phone (2005) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_code_switched_speech_corpus_7935e156,
  title = {code-switched speech corpus},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/code-switched-speech-corpus/rdr_7935e156e43246dab34b7b6b1abc8cb0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_code_switched_speech_corpus_7935e156_v1,
  title = {code-switched speech corpus},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/code-switched-speech-corpus/rdr_7935e156e43246dab34b7b6b1abc8cb0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake