In multilingual societies, speakers routinely weave multiple languages together within a single conversation or sentence—a phenomenon known as code-switching or code-mixing. For natural language processing (NLP) models, mastering code-switched text is notoriously difficult due to a chronic shortage of natural training data.
To bypass this "data drought," a common and intuitive heuristic is data augmentation via syntax-agnostic substitution: taking a monolingual sentence and mechanically replacing random words or phrases with their translations. While this approach is simple to implement, a growing body of research reveals a fundamental flaw: it creates a severe synthetic-to-natural data gap.
To understand why this gap exists, and how modern techniques aim to bridge it, we must look at how language models process the underlying structural rules of human speech.
The Illusion of Lexical Swapping
The core assumption behind simple word-for-word substitution is that code-switching is merely a lexical variation of a base language. However, language models do not just look at words in isolation; they capture deep, structural relationships.
Frances Adriana Laureano De Leon et al. exposed this mismatch by using diagnostic probes on pre-trained language models like mBERT and XLM-RoBERTa. They discovered that these models maintain consistent internal representations of syntax across natural code-switched languages, but their performance drops sharply when faced with "fake" or synthetically substituted text.
The models can essentially "tell" when a sentence is artificially constructed. Mechanical substitution breaks the structural integrity that language models rely on to generalize, meaning that models trained on naive synthetic data often fail when deployed on real-world social media text or conversational speech (Binu Jasim et al.; Cesa Salaam et al.).
Why Humans Don't Switch Randomly
Authentic code-switching is governed by complex linguistic, pragmatic, and social constraints rather than random insertion. When synthetic data generation ignores these rules, it introduces unnatural patterns that can actively mislead a model.
1. Syntactic Constraints
Languages have different word orders and grammatical structures. Randomly swapping a word can violate the syntactic norms of both languages. To address this, Grandee Lee et al. grounded their data augmentation in the Matrix Language Frame (MLF) theory, ensuring that the grammatical framework of the dominant language is respected. Without such linguistic constraints, models fail to learn the true structural alignments between languages with divergent syntax (Robert Litschko et al.; Chunming Hu et al.). Native speakers possess an intuitive sense of where a transition feels natural; random substitution lacks this grammatical intuition (Sin-En Lu et al.).
2. Pragmatic and Sociolinguistic Dynamics
Humans switch languages for specific reasons—such as emphasizing a point, shifting style, or conveying emotional nuance. Tom Potter et al. argue that mechanical replacement completely misses these pragmatic motivations, leading to data that lacks stylistic fidelity. Furthermore, in spoken language, the statistical patterns of when and how long a speaker switches are highly structured; ignoring these phonetic and temporal transitions prevents speech recognition models from generalizing to real speech (Karan Taneja et al.).
Modeling Fluid Transitions
To close the synthetic-to-natural gap, recent work shifts away from rigid, rule-based word swapping toward frameworks that dynamically learn how languages blend.
Adversarial and Contextual Generation
Rather than defining manual substitution rules, researchers use generative models to capture the natural flow of code-switching. Khyathi Chandu et al. framed code-switching as a style transfer problem, using a Generative Adversarial Network (GAN) to learn how to transition between languages based on a small sample of natural text. Similarly, Junlan Feng et al. combined BERT's bidirectional context with a GAN discriminator. BERT predicts appropriate foreign words to "fill in the blanks," while the discriminator ensures the resulting sentence maintains a natural, human-like flow.
Naive Substitution:
[English Base] -> "I want to eat an apple."
-> "I want to eat an [سیب]." (Grammatically/phonetically awkward)
Contextual/Generative Simulation:
[English Base] -> BERT/GAN Alignment -> "I want to eat [سیب]."
-> (Respects natural switch boundaries and determiner constraints)
Translation-Based Simulation
Another powerful approach is reframing data generation as a translation task. The "Translation for Code-Switching" (TCS) model developed by Ishan Tarunesh et al. adapts unsupervised machine translation to generate mixed text that is nearly indistinguishable from real-world conversations. To scale this without massive parallel data, Maksim Borisov et al. utilized contextual word alignment tools (like SimAlign) to identify high-confidence cross-lingual anchors, simulating realistic code-switching points within standard monolingual datasets.
Curricula and the Safety Blind Spot
Even when high-quality synthetic data is available, how it is introduced during training matters. Chunming Hu et al. introduced Progressive Code-Switching (PCS), a curriculum learning strategy that starts with simple word swaps and gradually introduces complex, task-critical terms. This allows the model to steadily build structural alignments without being overwhelmed by chaotic syntax early on. Similarly, Maab Elrashid et al. demonstrated that a domain curriculum combined with an encoder alignment loss prevents models from experiencing "catastrophic forgetting" when transitioning from monolingual to mixed-language tasks.
The Safety Risks of Artificial Mixtures
The synthetic-to-natural gap is not just an academic performance metric; it represents a critical vulnerability in Large Language Model (LLM) alignment. While modern LLMs are highly multilingual, their safety training is predominantly single-language.
When evaluating model safety or red-teaming vulnerabilities, relying on token-level random translation (Jiayang Song et al.) or basic model-generated synthesis (Haneul Yoo et al.) fails to uncover real-world exploits. Because these automated methods miss complex, culturally grounded structures like tag-sentential switching, they create a false sense of security. Adversaries using natural, fluid code-switching can easily bypass safety filters because the model's safety alignment was evaluated on artificial, rigid substitutions.
Summary
The transition from syntax-agnostic substitution to structurally and pragmatically aware generation marks a maturity milestone in multilingual NLP. By recognizing that code-switching is an intricate linguistic system rather than a random mixture of words, modern frameworks are successfully building models that understand not just multiple languages, but the organic ways humans choose to blend them.
Key Foundational Work
Style Variation as a Vantage Point for Code-Switching (2020) — contributed 1 concept elements to this direction.
Exploiting Monolingual Speech Corpora for Code-Mixed Speech Recognition (2019) — contributed 1 concept elements to this direction.
A Semi-supervised Approach to Generate the Code-Mixed Text using Pre-trained Encoder and Transfer Learning (2020) — contributed 1 concept elements to this direction.