TL;DR
This paper introduces MixSarc, the first publicly available dataset for identifying implicit meanings (humor, sarcasm, offense, vulgarity) in Bangla-English code-mixed social media text. They demonstrate that while humor is relatively detectable, sarcasm and other nuanced expressions remain challenging for current models, highlighting the need for culturally-aware NLP solutions.
What Problem Does This Paper Address?
Social media is a melting pot of languages, especially in regions like South Asia where code-mixing between languages like Bangla and English is common. This linguistic blend, often seen in informal online communication, poses a significant challenge for Natural Language Processing (NLP) systems. Existing NLP models, primarily trained on monolingual or high-resource languages, struggle to understand the subtle nuances of code-mixed text. This is particularly true for identifying implicit meanings like humor, sarcasm, offense, and vulgarity, which rely heavily on cultural context, informal spellings, and language switching within a single sentence. The paper aims to bridge this gap by creating a dedicated resource for this understudied area.
Methods
The authors developed a multi-stage pipeline to create the MixSarc dataset. This involved:
- Data Sourcing: Collecting posts from high-engagement Facebook pages known for informal, culture-specific Bangla-English content using the Scrapy framework.
- Data Cleaning and Preprocessing: Removing non-textual elements like emojis, filtering out posts not exhibiting genuine code-mixing (using a fine-tuned mBERT classifier), and discarding very short posts. A key criterion was that at least 30% of tokens in a post had to be classified as Banglish (Romanized Bangla).
- Annotation: A rigorous annotation process involving 18 native Bangla speakers who were fluent in both Bangla and English. Each sentence was independently annotated by three annotators for four binary labels: Humorous, Sarcastic, Offensive, and Vulgar. Final labels were determined by majority voting. The annotation scheme allowed for multi-label assignments, reflecting that a single sentence can exhibit multiple characteristics.
- Benchmarking: They established baseline performance using supervised transformer models (Banglish-BERT and Gemma-2B) and evaluated five state-of-the-art large language models (LLMs) in a zero-shot setting using structured prompts.
Datasets
The primary contribution is the MixSarc dataset, containing 9,087 manually annotated Bangla-English code-mixed sentences. These sentences are labeled for humor, sarcasm, offensiveness, and vulgarity. The dataset was split into training, validation, and testing sets using a 70:15:15 ratio with stratified sampling.
The authors also used the BnSentMix dataset for a case study to demonstrate the impact of sarcasm detection on sentiment analysis.
Key Insights and Ideas
- The "Implicit Meaning" Challenge: Detecting humor, sarcasm, offense, and vulgarity in code-mixed text is significantly harder than standard sentiment analysis. These phenomena are deeply intertwined with cultural context and subtle linguistic cues.
- Humor is the Easiest: Among the four implicit meanings, humor was the most reliably detected by both supervised models and zero-shot LLMs, suggesting it has more discernible linguistic markers in this context.
- Sarcasm is a Major Hurdle: Sarcasm, in particular, proved to be a substantial challenge. The paper highlights that over 42% of negative sentiment instances in an external dataset (BnSentMix) exhibited sarcastic characteristics, indicating that many sentiment analysis systems might be misinterpreting genuine negativity due to unaddressed sarcasm.
- Class Imbalance is a Killer: The rarity of offensive and vulgar content in the dataset led to extremely low recall and F1-scores for these categories, even for advanced LLMs. This underscores the critical need for techniques that address severe class imbalance in NLP tasks.
- Zero-Shot LLMs as Baselines: While not outperforming fine-tuned models, zero-shot LLMs provide valuable training-free baselines and demonstrate a foundational understanding of implicit meanings, especially for humor. However, their performance on minority classes and sarcasm is limited.
Key Results
- MixSarc Dataset: Contains 9,087 annotated sentences, with 4,041 labeled as humorous and 1,475 as sarcastic. Offensive and vulgar labels are less frequent (207 and 290, respectively).
- Supervised Models: Banglish-BERT achieved an F1-score of 0.708 for Humor. For Sarcasm, the F1-score dropped to 0.3938. Offense and Vulgarity saw very low recall and F1-scores due to class imbalance.
- Zero-Shot LLMs: Performed best on Humor, with Claude Opus 4.8 and GPT-5.5 achieving F1-scores around 0.6294. LLaMA 4 Maverick showed the highest sarcasm F1-score (0.3905) but with very low recall for other models. Claude Opus 4.8 led in Offensive and Vulgar categories with F1-scores of 0.3206 and 0.3784, respectively.
- Sarcasm Impact on Sentiment: A MixSarc-tuned classifier estimated that approximately 42.13% of negative sentiment samples in BnSentMix were sarcastic.
Limitations and Future Work
- Class Imbalance: The authors acknowledge that the rarity of offensive and vulgar content significantly hinders model performance.
- Annotation Subjectivity: The inherent subjectivity of humor, sarcasm, and offense, coupled with annotator diversity, introduces variability in annotations.
- Future Work: The paper suggests the need for:
- Better class balancing techniques.
- Domain-specific augmentation strategies.
- More context-aware architectures for detecting minority classes.
- Further research into culturally aware and pragmatics-aware NLP models for low-resource multilingual settings.
- The MixSarc dataset is intended to be a foundational resource for future research in this area.
Summary updated