Disentangling Codemixing in Chats: The NUS ABC Codemixed Corpus

Svetlana Churina, Akshat Gupta, Insyirah Mujtahid, Kokil Jaidka

Introduction

Researchers from the National University of Singapore have introduced the ABC (Air Batu Campur) Codemixed Corpus, a substantial dataset comprising over 355,000 private instant messages. Unlike existing datasets typically scraped from public social media platforms, this corpus is uniquely author-labeled with relationship metadata. This allows for a granular study of "code-mixing"—the practice of switching between languages such as English, Mandarin, and Malay—and how it serves as a tool for managing intimacy and social dynamics in private digital spheres.

The Challenge of Private Codemixing

Current Natural Language Processing (NLP) research on code-mixing predominantly relies on "performative" data from public forums like Twitter or Reddit. However, linguistic behavior in public spaces differs significantly from private interactions with close friends or colleagues.

A critical gap exists: the lack of high-quality, large-scale corpora of private conversations that include both linguistic data and social context, such as the degree of intimacy between speakers. Without this context, developing models that understand the nuances of multilingual communication—such as using a specific dialect to soften a request or express frustration—remains a challenge.

Methodology and Pipeline

The study utilized a "voluntary data donation" framework, involving 166 participants who contributed 477 chat threads from platforms like Telegram and WhatsApp. To manage the inherent "messiness" of real-world chat data, the researchers developed a sophisticated processing pipeline.

The data processing pipeline from original chat data to language-tagged and translated output.

The pipeline includes:

  • Preprocessing: Standard cleaning (removing PII, URLs, and timestamps) while preserving emojis and casing to maintain linguistic signals.
  • Language Identification (LID): Moving beyond standard models like fastText, the researchers employed Qwen2.5-14B-Instruct for token-level tagging. This approach proved superior in identifying "Singlish" (Singaporean English) particles and transliterated words often missed by traditional models.
  • Translation: The same LLM was used to translate entire conversations into English. Conversation-level context was found to be essential, as message-level translation often failed due to a lack of situational context.
  • Analysis: The data was analyzed through statistical patterns, linguistic features (using LIWC-22), and social functions based on the Systemic Functional Linguistics framework.

The NUS ABC Corpus

The resulting NUS ABC Codemixed Corpus contains 355,641 messages spanning English, Mandarin, Hokkien, Malay, and Tamil. A key feature is the inclusion of metadata regarding relationship types (Close, General, Work) and self-reported intimacy scores, providing the necessary social context for linguistic analysis.

Demographics, motivations for social media use, and the intimacy/disclosure factors used to categorize relationships in the corpus.

Key Findings on Intimacy and Language

The analysis revealed several significant insights:

  • Intimacy Drives Mixing: Code-mixing frequency is highest in short messages and close relationships. In intimate conversations, the mixing rate for short messages (0–5 tokens) reaches 36.7%, notably higher than in professional exchanges.
  • LLMs as Superior Annotators: Qwen2.5 outperformed traditional LID models for "low-resource" informal varieties, excelling at identifying irregular English (slang/abbreviations) and transliterated Asian languages.
  • The "Tenor" of Chat: Using Systemic Functional Linguistics, code-mixing was categorized into three "tenors":
  • Affective: Using markers like sayang (Malay for "dear") or jialat (Hokkien for "troublesome") to manage emotions.
  • Interpersonal-Social: Using markers like paiseh (Hokkien for "embarrassed/sorry") to maintain politeness and rapport.
  • Discourse-Stylistic: Using particles like ah, hor, meh to manage conversation flow and signal group identity.

Conclusion and Future Directions

The NUS ABC Corpus represents a significant step forward in understanding private, multilingual digital communication. While the current dataset is primarily composed of Singaporean university students, it provides a robust foundation for future research. The authors aim to expand the corpus to include more diverse demographics and explore dialect-specific adaptation for NLP models, ultimately building systems that better reflect the complexity of human social interaction.

Summary updated

BibTeX
@inproceedings{churina2026disentangling,
  author = {Svetlana Churina and Akshat Gupta and Insyirah Mujtahid and Kokil Jaidka},
  title = {{Disentangling Codemixing in Chats: The NUS ABC Codemixed Corpus}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.80},
}
Figure