When a user posts on social media in a multilingual community, they rarely restrict themselves to a single language or a single script. Instead, they might write a Tamil phrase using the Roman alphabet, inject English nouns, and apply local slang. This phenomenon—code-mixing accompanied by phonetic transliteration—is the default mode of digital communication for billions of people worldwide.
Yet, modern Natural Language Processing (NLP) models frequently break down when encountering it. Large language models that excel at complex reasoning in English or Mandarin struggle to perform basic sentiment analysis or hate speech detection on code-mixed text. To understand why this happens, we have to look at how these models process text, how code-mixing alters grammar, and how researchers are redesigning architectures to bridge the gap.
The Tokenization Trap
To understand the failure of state-of-the-art models on code-mixed text, we must first look at subword tokenization (such as Byte-Pair Encoding or WordPiece).
Pre-trained multilingual models like mBERT or XLM-RoBERTa are trained on massive monolingual corpora. They learn English written in the Latin script, and they learn Hindi written in the Devanagari script. However, when a social media user writes a Hindi word like achha (meaning "good") using the Latin script, the tokenizer encounters a sequence it was never explicitly taught.
Instead of recognizing achha as a single semantic unit, the tokenizer fragments it into arbitrary subwords (e.g., [ac, hha]). This fragmentation dilutes the semantic signal. As Vinay Gopalan and Mark Hopkins demonstrate in Reed at SemEval-2020 Task 9: Fine-Tuning and Bag-of-Words Approaches to Code-Mixed Sentiment Analysis, this exact tokenization mismatch prevents large pre-trained language models from deeply understanding code-mixed text. In fact, the authors found that simpler, traditional bag-of-words models can achieve surprisingly competitive results because they do not suffer from the structural confusion introduced by deep subword tokenizers on unseen script combinations.
The Emergence of a Third Code
Code-mixing is not merely the mechanical concatenation of two languages at sentence boundaries. Instead, it occurs intra-sententially, creating a distinct hybrid linguistic ecosystem.
In a study on postcolonial literature, Yan Yan describes this fusion as a "third code"—a new linguistic space that merges the structural and cultural properties of different dialects and languages (CHINESE TRANSLATION OF CULTURE-LOADED WORDS IN AFRICAN ENGLISH LITERATURE FROM THE PERSPECTIVE OF ECO-TRANSLATOLOGY: A CASE STUDY OF GURNAH’S BY THE SEA).
When applied to digital spaces, this third code introduces non-standard grammar and fluid syntax that standard monolingual models cannot parse. As noted by Konthala Yasaswini et al. in IIITT@DravidianLangTech-EACL2021: Transfer Learning for Offensive Language Detection in Dravidian Languages, this structural variability severely hinders a model's ability to generalize. Because the transitions between languages are informal and dynamic, standard models fail to maintain syntactic coherence, making tasks like offensive language detection in Dravidian languages exceptionally difficult.
Phonetic Volatility and Spelling Variations
Compounding the tokenization and syntactic challenges is the complete lack of standardized orthography. When writing phonetically, spelling becomes entirely subjective. The word achha might be spelled acha, achha, achhaas, or achaas by different users within the same comment thread.
This phonetic volatility creates massive vocabulary inflation and noise. In TueMix at SemEval-2020 Task 9: Logistic Regression with Linguistic Feature Set, Elizabeth Bear et al. highlight how these non-standard variations cause deep neural networks to struggle. When spelling is unpredictable, the model cannot reliably map words to stable vector representations. In such environments, the authors found that explicitly engineering linguistic feature sets can outperform standard neural text representations by capturing the underlying patterns of language switching directly.
Adapting Models to Hybrid Spaces
To resolve these challenges, researchers have developed three primary paradigms to adapt machine learning models to code-mixed text: Lexical Normalization, Transliteration-Aware Pre-training, and Representation Alignment.
Approach 1: Lexical Normalization
[Volatile Input Text] ──> [Seq2Seq + Levenshtein] ──> [Standardized Text] ──> [Standard NLP Model]
Approach 2: Transliteration-Aware Pre-training
[Transliterated Corpus] ──> [Masked Language Modeling] ──> [Script-Agnostic Transformer]
Approach 3: Representation Alignment
[Code-Mixed Text] ──┐
├──> [Contrastive Loss] ──> [Unified Embedding Space]
[Monolingual Text] ─┘
1. Lexical Normalization
One intuitive solution is to clean the text before it ever reaches the classifier. By mapping volatile phonetic variations back to a canonical form, we can restore the performance of standard models.
For instance, Soumil Mandal and Karthick Nanmaran developed a two-stage architecture to handle this normalization (Normalization of Transliterated Words in Code-Mixed Data Using Seq2Seq Model & Levenshtein Distance). Their system uses a character-level sequence-to-sequence model to generate initial standard candidates, which are then filtered using Levenshtein distance against a standardized lexicon. This effectively strips away the phonetic noise of social media text.
2. Transliteration-Aware Pre-training
Instead of forcing the text to conform to the model, we can force the model to understand the text's native environment. This involves pre-training or fine-tuning language models directly on transliterated and mixed-script corpora using custom objectives.
In byteSizedLLM@DravidianLangTech 2025: Fake News Detection in Dravidian Languages Using Transliteration-Aware XLM-RoBERTa and Attention-BiLSTM, Rohith Gowtham Kodali and Durga Prasad Manukonda utilized a transliteration-aware Masked Language Modeling (MLM) approach. By exposing an XLM-RoBERTa model to Malayalam text written in both its native script and Romanized forms during training, the model learns to bridge the script divide, allowing an attention-enhanced BiLSTM downstream classifier to accurately detect fake news despite script diversity.
3. Representation Alignment via Contrastive Learning
A third approach bypasses explicit normalization by forcing the model's internal vector space to treat code-mixed text and its monolingual equivalents as semantically identical.
In BpHigh at WASSA 2023: Using Contrastive Learning to build Sentence Transformer models for Multi-Class Emotion Classification in Code-mixed Urdu, Bhavish Pahwa leveraged unsupervised contrastive learning to build robust sentence transformers for code-mixed Urdu. By pulling semantically similar representations closer together and pushing dissimilar ones apart in the embedding space, the model learns to ignore surface-level script variations and focus entirely on the underlying emotional and semantic signals.