On social media, multilingual individuals rarely restrict themselves to a single language or script. Instead, they seamlessly blend multiple languages within a single post—a phenomenon known as code-mixing. For example, a user might combine Hindi and English vocabulary ("Hinglish") or mix Tamil with English. Compounding this complexity, users frequently employ transliteration, writing non-English words using the Roman alphabet (e.g., typing the Hindi word achha instead of अच्छा).
Traditional Natural Language Processing (NLP) models are fundamentally ill-equipped for this linguistic fluidity. They are typically trained on clean, monolingual corpora written in native scripts. When confronted with code-mixed, transliterated text, standard tokenizers fragment words into meaningless sub-words, and embedding spaces fail to capture the underlying sentiment.
To solve this, researchers have developed specialized strategies that balance the structural robustness of traditional machine learning with the contextual awareness of deep learning.
Why Traditional Classifiers Remain Surprisingly Competitive
In standard NLP, deep learning and transformer architectures have largely rendered traditional machine learning obsolete. However, in the domain of code-mixed social media text, simpler models like Logistic Regression, Support Vector Machines (SVMs), and Decision Trees frequently match or outperform deep neural networks.
To understand why, consider how a transformer model processes text versus how a character-level machine learning model does. Transformers rely on sub-word tokenization algorithms (like Byte-Pair Encoding) trained on massive monolingual datasets. When a user transliterates a Dravidian language into Roman script, the spelling is highly irregular; the same word might be spelled three different ways by three different users. A transformer's tokenizer will split these variations into completely different, arbitrary sub-word tokens, destroying the semantic connection between them.
In contrast, traditional models leveraging character n-grams or explicit linguistic features bypass this fragmentation:
- Capturing Morphological Invariance: Character n-grams look at overlapping substrings of characters. If a word's spelling varies slightly due to informal transliteration, character n-grams still capture the shared structural core of the word. As demonstrated by Sunil Saumya et al. in their work on offensive language identification, traditional machine learning models with character n-gram features surprisingly outperformed complex deep learning and transfer learning models in code-mixed Dravidian text.
- Leveraging Explicit Linguistic Signals: Instead of forcing a neural network to implicitly learn the boundaries between languages, traditional classifiers can be fed explicit token-level language identifiers and engineered linguistic features. For instance, Elizabeth Bear et al. developed a logistic regression model for the SemEval-2020 SentiMix task that significantly outperformed neural network approaches by incorporating targeted linguistic feature sets.
- Mitigating Data Scarcity: Deep learning models require vast amounts of data to generalize. When data is scarce and highly imbalanced—a common issue in low-resource languages—simpler models are less prone to overfitting. Vinay Gopalan et al. found that simpler bag-of-words models achieved highly competitive results compared to fine-tuned BERT models, suggesting that standard pre-trained models often lack a deep, native understanding of code-mixed syntax.
The Power and Limits of Cross-Lingual Transformers
Despite the resilience of traditional classifiers, large pre-trained language models and transformers possess a distinct advantage: they excel at capturing long-range context and implicit semantic nuances, such as sarcasm or mixed feelings.
When transformers do succeed in code-mixed environments, it is typically because they leverage multilingual pre-training (like mBERT or RoBERTa). These models map words from different languages into a shared semantic space. For example, if the model understands the English word "happy" and the Hindi word "खुश" (khush), its internal representations can help bridge the gap when it encounters the transliterated "khush" in a code-mixed sentence, provided it receives adequate fine-tuning.
The literature reflects a clear tension between these two paradigms:
┌─────────────────────────────────────────────────────────────────┐
│ THE MODELING TENSION │
├────────────────────────────────┬────────────────────────────────┤
│ Traditional ML Classifiers │ Transformer Models │
├────────────────────────────────┼────────────────────────────────┤
│ • Robust to spelling variations│ • Captures deep context & │
│ via character n-grams. │ subtle emotional nuances. │
│ • Performs well with limited, │ • Leverages cross-lingual │
│ imbalanced data. │ knowledge transfer. │
│ • Misses complex context. │ • Vulnerable to sub-word │
│ │ tokenization fragmentation. │
└────────────────────────────────┴────────────────────────────────┘
For tasks requiring deep emotional comprehension, transformers show clear superiority. Anshul Wadhawan et al. demonstrated that transformer-based models significantly outperform traditional deep learning for emotion recognition in Hindi-English code-mixed data, where identifying specific emotional states requires a holistic understanding of the sentence structure. Similarly, Apurva Parikh et al. showed that a transfer learning approach using BERT significantly outperformed traditional TF-IDF methods on Hindi-English tweets.
However, the challenge of generalization remains steep. Because social media text is highly dynamic and plagued by severe class imbalances (often skewing heavily positive or negative depending on the platform), complex architectures can easily overfit. This vulnerability was highlighted by Anusha M D Gowda et al., whose deep learning hybrid meta-model suffered from significant overfitting when moving from the development set to real-world Tamil code-mixed test data.
Grounding Models with Specialized Corpora
Because code-mixing patterns are highly regional, cultural, and platform-dependent, general-purpose multilingual models cannot solve this problem out of the box. The foundation of progress in this field relies on the curation of specialized, token-level annotated datasets that reflect how communities actually communicate online.
Recent efforts have focused on expanding resources beyond major language pairs to encompass a wider variety of under-represented linguistic phenomena:
- Dravidian Languages: Datasets like KanCMD, introduced by Adeep Hande et al., provide Kannada-English YouTube comments annotated for both sentiment and offensive language. Other efforts have targeted Tamil and Malayalam, as well as highly resource-constrained languages like Tulu.
- Bengali: The BnSentMix dataset, developed by Sadia Alam et al., offers a large-scale corpus of 20,000 Bengali-English samples, introducing a dedicated "mixed" sentiment category to capture sentences where positive and negative sentiments are intertwined across language boundaries.
- Amerindian-European Blends: Moving beyond South Asian languages, Marvin Agüero-Torales et al. introduced the JOSA dataset to evaluate sentiment analysis in Jopara, a code-switching language that blends Guarani and Spanish.
By providing explicit sentiment labels alongside word-level language tags, these datasets allow models to learn exactly where a language switch occurs and how that switch influences the emotional trajectory of the text. Whether through robust character-level machine learning or context-aware transformers, mastering code-mixed sentiment analysis ultimately depends on training models to embrace, rather than ignore, the linguistic hybridity of modern digital speech.
Key Foundational Work
IRLab_DAIICT at SemEval-2020 Task 9: Machine Learning and Deep Learning Methods for Sentiment Analysis of Code-Mixed Tweets (2020) — contributed 1 concept elements to this direction.
KanCMD: Kannada CodeMixed Dataset for Sentiment Analysis and Offensive Language Detection (2020) — contributed 1 concept elements to this direction.
JUNLP at SemEval-2020 Task 9: Sentiment Analysis of Hindi-English Code Mixed Data Using Grid Search Cross Validation (2020) — contributed 1 concept elements to this direction.