Emotion Recognition in Code-Mixed Conversations

26 concepts19 papers1 proposal

In bilingual and multilingual communities, speakers frequently alternate between languages within a single conversation—a phenomenon known as code-mixing. For example, "Hinglish" fluidly blends Hindi and English, while code-mixed Urdu combines Urdu vocabulary with Latin script and English phrases.

While natural for humans, code-mixed dialogues present a severe challenge for Emotion Recognition in Conversation (ERC) systems. Traditional natural language processing (NLP) models are typically trained on monolingual data. When confronted with code-mixed text, they struggle with non-standard syntax, phonetic transliterations, varied spellings, and highly context-dependent emotional expressions. To build empathetic conversational AI, researchers have developed specialized strategies to capture emotional nuances across shifting linguistic boundaries.

The Challenge of Fluid Language Switching

Why do standard emotion recognition models fail on code-mixed text? The primary issue is semantic fragmentation. When a speaker switches languages mid-sentence, the structural and contextual cues that monolingual models rely on are disrupted. Furthermore, social media and chat messages often include non-literal cues like emoticons, typos, and colloquialisms that obscure standard linguistic forms.

As noted by Rohith Arumugam S et al. in SSN_ARMM at SemEval-2024 Task 10: Emotion Detection in Multilingual Code-Mixed Conversations using LinearSVC and TF-IDF, traditional NLP models lack the capacity to handle the combined linguistic diversity and cultural nuances inherent in these interactions. To overcome this, current research splits into three primary methodologies: translation to high-resource languages, native multilingual modeling, and grounding via external lexicons or commonsense knowledge.

Strategy 1: Translation and High-Resource Alignment

One intuitive way to resolve code-mixing complexity is to normalize the text by translating it entirely into a high-resource language like English.

In Hidetsune at SemEval-2024 Task 10: An English Based Approach to Emotion Recognition in Hindi-English code-mixed Conversations Using Machine Learning and Machine Translation, Hidetsune Takahashi demonstrates that translating mixed-language text into English allows developers to leverage vast, pre-existing monolingual emotion datasets and established NLP techniques. Similarly, Marco Siino utilizes this pipeline in TransMistral at SemEval-2024 Task 10: Using Mistral 7B for Emotion Discovery and Reasoning its Flip in Conversation, applying a few-shot learning approach with the Mistral 7B large language model on the translated text.

Why it works: Translation maps irregular, low-resource code-mixed tokens into a well-structured, dense semantic space. By converting the text to English, the system can tap into the advanced reasoning capabilities of large language models that have been heavily optimized on English corpora, bypassing the need to learn the syntax of the mixed language from scratch.

Strategy 2: Native Multilingual Representation

Rather than relying on translation—which can strip away cultural nuances and emotional subtext unique to code-switching—many researchers fine-tune multilingual transformer models to process the mixture natively.

Models like mBERT, indicBERT, and MuRIL are pre-trained on multiple languages and can be adapted to recognize emotional patterns directly from code-mixed inputs. For instance, Antony Rajesh A et al. in SSN_Semeval10 at SemEval-2024 Task 10: Emotion Discovery and Reasoning its Flip in Conversations fine-tuned mBERT and indicBERT to capture semantic content alongside cultural interplay. For code-mixed Urdu, Bhavish Pahwa implemented unsupervised contrastive learning with the MuRIL model in BpHigh at WASSA 2023: Using Contrastive Learning to build Sentence Transformer models for Multi-Class Emotion Classification in Code-mixed Urdu to generate robust sentence embeddings.

To maximize performance, researchers frequently deploy ensembles or compare architectures. Dilip Venkatesh et al. evaluated both BERT and Llama 2 in BITS Pilani at SemEval-2024 Task 10: Fine-tuning BERT and Llama 2 for Emotion Recognition in Conversation, while Ankit Vaidya et al. built a transformer-based ensemble in CLTeam1 at SemEval-2024 Task 10: Large Language Model based ensemble for Emotion Detection in Hinglish. Even simpler architectures can capture sequential dependencies; Ravindran V et al. utilized a Bidirectional LSTM approach in TECHSSN at SemEval-2024 Task 10: LSTM-based Approach for Emotion Detection in Multilingual Code-Mixed Conversations.

Why it works: Multilingual pre-training and contrastive learning align the vector representations of different languages into a shared space. This alignment allows the model to recognize that a phrase in Hindi and its equivalent in English carry identical emotional weight, enabling the network to maintain a coherent emotional context even when vocabulary switches mid-utterance.

Strategy 3: Grounding with Lexicons and Commonsense

Because text alone can be ambiguous, grounding models with external psychological or commonsense knowledge provides language-agnostic anchors for emotion detection.

In VerbaNexAI Lab at SemEval-2024 Task 10: Emotion recognition and reasoning in mixed-coded conversations based on an NRC VAD approach, Santiago Garcia et al. integrated Valence, Arousal, and Dominance (VAD) scores from the NRC VAD Lexicon into Transformer and GRU models. Taking a different approach to external grounding, Shivani Kumar et al. introduced the COFFEE framework in From Multilingual Complexity to Emotional Clarity: Leveraging Commonsense to Unveil Emotions in Code-Mixed Dialogues, which extracts English-based commonsense knowledge to clarify the emotional intent behind multilingual speech. Non-verbal elements can also be mapped to text; Jinghui Zhang et al. utilized soft prompt tuning in Emotion classification on code-mixed text messages via soft prompt tuning to transform emoticons directly into textual descriptors.

[Code-Mixed Input] ──> [Multilingual Model] ──> [Ambiguous Representation]
                                                          │
[External Knowledge] ─> [VAD Lexicons / Commonsense] ─────┴─> [Stable Emotional Output]

Why it works: Psychological dimensions (like arousal) and commonsense truths remain stable regardless of the language spoken. By injecting these features, the model does not have to infer emotional states purely from volatile, code-mixed surface text; instead, it uses the external knowledge as an invariant anchor to decode intent.

Tracking Emotional Shifts and Triggers

Conversations are inherently dynamic, and recent benchmarks have pushed the field beyond static classification toward emotion reasoning. This involves identifying "emotion flips"—points where a speaker's emotional state shifts—and pinpointing the exact conversational turn that triggered the change.

This paradigm is central to the EDiReF (Emotion Discovery and Reasoning its Flip) framework explored by Mehrzad Tareh et al. in IASBS at SemEval-2024 Task 10: Delving into Emotion Discovery and Reasoning in Code-Mixed Conversations. To model these sequential dynamics, researchers combine contextual embeddings with sequence-aware architectures. Neng Wan et al. deployed a combination of BERT and GRUs in UCSC NLP at SemEval-2024 Task 10: Emotion Discovery and Reasoning its Flip in Conversation (EDiReF), while Ronghao Pan et al. fine-tuned sequential Transformers in UMUTeam at SemEval-2024 Task 10: Discovering and Reasoning about Emotions in Conversation using Transformers. Interestingly, Claudiu Creanga et al. found in ISDS-NLP at SemEval-2024 Task 10: Transformer based neural networks for emotion recognition in conversations that fine-tuned Masked Language Models (MLMs) often outperform large generative models when tracking these conversational flips.

Despite these architectural innovations, a substantial performance gap remains between monolingual baseline tasks and code-mixed conversational reasoning. As noted by Sara Niță et al. in RACAI at SemEval-2024 Task 10: Combining algorithms for code-mixed Emotion Recognition in Conversation, combining traditional machine learning with deep learning models yields mixed results, highlighting that simply scaling models is insufficient. Future progress relies on developing architectures that natively integrate conversational context, cross-lingual embeddings, and cultural common sense.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_emotion_recognition_in_code_mixed_conver_9fc1a290,
  title = {emotion recognition in code-mixed conversations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/emotion-recognition-in-code-mixed-conversations/rdr_9fc1a29014b6474cbf789a5403f52415},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_emotion_recognition_in_code_mixed_conver_9fc1a290_v1,
  title = {emotion recognition in code-mixed conversations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/emotion-recognition-in-code-mixed-conversations/rdr_9fc1a29014b6474cbf789a5403f52415/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake