MixSarc: A Bangla-English Code-Mixed Corpus for Implicit Meaning Identification

Kazi Samin Yasar Alam, Md. Waqeeb T. S. Chowdhury, Tamim Ahmed, Ajwad Abrar, Md Rafid Haque

TL;DR

This paper introduces MixSarc, the first publicly available dataset for identifying implicit meanings (humor, sarcasm, offense, vulgarity) in Bangla-English code-mixed social media text. They demonstrate that while humor is relatively detectable, sarcasm and other nuanced expressions remain challenging for current models, highlighting the need for culturally-aware NLP solutions.

What Problem Does This Paper Address?

Social media is a melting pot of languages, especially in regions like South Asia where code-mixing between languages like Bangla and English is common. This linguistic blend, often seen in informal online communication, poses a significant challenge for Natural Language Processing (NLP) systems. Existing NLP models, primarily trained on monolingual or high-resource languages, struggle to understand the subtle nuances of code-mixed text. This is particularly true for identifying implicit meanings like humor, sarcasm, offense, and vulgarity, which rely heavily on cultural context, informal spellings, and language switching within a single sentence. The paper aims to bridge this gap by creating a dedicated resource for this understudied area.

Methods

The authors developed a multi-stage pipeline to create the MixSarc dataset. This involved:

  • Data Sourcing: Collecting posts from high-engagement Facebook pages known for informal, culture-specific Bangla-English content using the Scrapy framework.
  • Data Cleaning and Preprocessing: Removing non-textual elements like emojis, filtering out posts not exhibiting genuine code-mixing (using a fine-tuned mBERT classifier), and discarding very short posts. A key criterion was that at least 30% of tokens in a post had to be classified as Banglish (Romanized Bangla).
  • Annotation: A rigorous annotation process involving 18 native Bangla speakers who were fluent in both Bangla and English. Each sentence was independently annotated by three annotators for four binary labels: Humorous, Sarcastic, Offensive, and Vulgar. Final labels were determined by majority voting. The annotation scheme allowed for multi-label assignments, reflecting that a single sentence can exhibit multiple characteristics.
  • Benchmarking: They established baseline performance using supervised transformer models (Banglish-BERT and Gemma-2B) and evaluated five state-of-the-art large language models (LLMs) in a zero-shot setting using structured prompts.

Datasets

The primary contribution is the MixSarc dataset, containing 9,087 manually annotated Bangla-English code-mixed sentences. These sentences are labeled for humor, sarcasm, offensiveness, and vulgarity. The dataset was split into training, validation, and testing sets using a 70:15:15 ratio with stratified sampling.

The authors also used the BnSentMix dataset for a case study to demonstrate the impact of sarcasm detection on sentiment analysis.

Key Insights and Ideas

  • The "Implicit Meaning" Challenge: Detecting humor, sarcasm, offense, and vulgarity in code-mixed text is significantly harder than standard sentiment analysis. These phenomena are deeply intertwined with cultural context and subtle linguistic cues.
  • Humor is the Easiest: Among the four implicit meanings, humor was the most reliably detected by both supervised models and zero-shot LLMs, suggesting it has more discernible linguistic markers in this context.
  • Sarcasm is a Major Hurdle: Sarcasm, in particular, proved to be a substantial challenge. The paper highlights that over 42% of negative sentiment instances in an external dataset (BnSentMix) exhibited sarcastic characteristics, indicating that many sentiment analysis systems might be misinterpreting genuine negativity due to unaddressed sarcasm.
  • Class Imbalance is a Killer: The rarity of offensive and vulgar content in the dataset led to extremely low recall and F1-scores for these categories, even for advanced LLMs. This underscores the critical need for techniques that address severe class imbalance in NLP tasks.
  • Zero-Shot LLMs as Baselines: While not outperforming fine-tuned models, zero-shot LLMs provide valuable training-free baselines and demonstrate a foundational understanding of implicit meanings, especially for humor. However, their performance on minority classes and sarcasm is limited.

Key Results

  • MixSarc Dataset: Contains 9,087 annotated sentences, with 4,041 labeled as humorous and 1,475 as sarcastic. Offensive and vulgar labels are less frequent (207 and 290, respectively).
  • Supervised Models: Banglish-BERT achieved an F1-score of 0.708 for Humor. For Sarcasm, the F1-score dropped to 0.3938. Offense and Vulgarity saw very low recall and F1-scores due to class imbalance.
  • Zero-Shot LLMs: Performed best on Humor, with Claude Opus 4.8 and GPT-5.5 achieving F1-scores around 0.6294. LLaMA 4 Maverick showed the highest sarcasm F1-score (0.3905) but with very low recall for other models. Claude Opus 4.8 led in Offensive and Vulgar categories with F1-scores of 0.3206 and 0.3784, respectively.
  • Sarcasm Impact on Sentiment: A MixSarc-tuned classifier estimated that approximately 42.13% of negative sentiment samples in BnSentMix were sarcastic.

Limitations and Future Work

  • Class Imbalance: The authors acknowledge that the rarity of offensive and vulgar content significantly hinders model performance.
  • Annotation Subjectivity: The inherent subjectivity of humor, sarcasm, and offense, coupled with annotator diversity, introduces variability in annotations.
  • Future Work: The paper suggests the need for:
  • Better class balancing techniques.
  • Domain-specific augmentation strategies.
  • More context-aware architectures for detecting minority classes.
  • Further research into culturally aware and pragmatics-aware NLP models for low-resource multilingual settings.
  • The MixSarc dataset is intended to be a foundational resource for future research in this area.

Summary updated

Abstract

Bangla-English code-mixing is widespread across South Asian social media, yet resources for implicit meaning identification in this setting remain scarce. Existing sentiment and sarcasm models largely focus on monolingual English or high-resource languages and struggle with transliteration variation, cultural references, and intra-sentential language switching. To address this gap, we introduce MixSarc, the first publicly available Bangla-English code-mixed corpus for implicit meaning identification. The dataset contains 9,087 manually annotated sentences labeled for humor, sarcasm, offensiveness, and vulgarity. We construct the corpus through targeted social media collection, systematic filtering, and multi-annotator validation. We benchmark transformer-based models and evaluate zero-shot large language models under structured prompting. Results show strong performance on humor detection but substantial degradation on sarcasm, offense, and vulgarity due to class imbalance and pragmatic complexity. Zero-shot models achieve competitive micro-F1 scores but low exact match accuracy. Further analysis reveals that over 42\% of negative sentiment instances in an external dataset exhibit sarcastic characteristics. MixSarc provides a foundational resource for culturally aware NLP and supports more reliable multi-label modeling in code-mixed environments.
BibTeX
@misc{alam2026mixsarc,
  author = {Kazi Samin Yasar Alam and Md. Waqeeb T. S. Chowdhury and Tamim Ahmed and Ajwad Abrar and Md Rafid Haque},
  title = {{MixSarc: A Bangla-English Code-Mixed Corpus for Implicit Meaning Identification}},
  year = {2026},
  eprint = {2602.21608},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.21608},
}
Figure