Introduction
Social media platforms were intended to be digital "town squares," but they often devolve into spaces rife with abuse. While automated detection models work reasonably well for English, they frequently fail for Indic languages—spoken by over a sixth of the global population.
To address this, researchers from ShareChat and IIT Kharagpur have released MACD, the largest human-annotated dataset for abusive speech in Indic languages, containing 150,000 comments across five languages. Alongside the dataset, they introduce AbuseXLMR, a domain-adapted transformer model pretrained on 5 million social media comments, which significantly outperforms general-purpose models in detecting toxicity within the nuanced, informal world of Indic social media.
The Challenge
The difficulty in moderating Indic social media stems from two primary issues:
- Data Scarcity: Existing Indic datasets are small (often under 6,000 samples) and lack linguistic diversity.
- Domain Gap: Standard multilingual models like mBERT or XLM-R are trained on formal text corpora such as Wikipedia. They struggle to parse the "wild west" of social media, which is characterized by spelling mistakes, slang, grammatical errors, and subtle, implicit insults that avoid standard trigger words.
Methodology: AbuseXLMR and Annotation
Instead of training a model from scratch, the researchers developed AbuseXLMR by taking the XLM-R architecture and performing "continued pretraining" using Masked Language Modeling (MLM) on a massive corpus of 5 million unlabeled social media comments from ShareChat.
XLM-R was chosen over alternatives like MuRIL because it does not require "Next Sentence Prediction" during training. Social media comments are typically isolated sentences; forcing a model to find coherence between unrelated comments can actually hinder its performance in this domain.
To build the accompanying dataset, the team established rigorous annotation guidelines. They categorized comments into specific types, such as profanity, sexual references, religious/political hate, and "implicit abuse" (harmful intent without explicit slurs), ensuring a nuanced understanding of platform toxicity.
The MACD Dataset
The paper introduces the Multilingual Abusive Comment Detection (MACD) dataset, which sets a new standard for scale and quality in Indic language moderation:
- Scale & Diversity: It contains 150,000 human-annotated comments covering five major Indic languages: Hindi, Tamil, Telugu, Malayalam, and Kannada.
- Balance: The dataset is carefully curated to be nearly balanced (49% abusive, 51% non-abusive). This is a significant achievement, as real-world abuse is usually a "needle in a haystack," making balanced training data hard to source.
- Metadata: It includes anonymized mappings between comments, posts, and users, enabling future research into social graph relationships and coordinated abuse.
Key Findings and Results
- State-of-the-Art Performance: AbuseXLMR consistently beat general-purpose models like XLM-R and MuRIL across all five languages in the MACD dataset, often by several percentage points in F1-macro score.
- Bridging the Domain Gap: By exposing AbuseXLMR to 5 million real-world comments, the model successfully learned the specific nuances of how people actually communicate on Indic platforms, proving that general-purpose models are often "too polite" for social media.
- Cross-Learning Benefits: "Joint training"—training on all five languages simultaneously—improved performance over training on a single language. This suggests the model learns universal patterns of abuse that transcend specific linguistic boundaries.
- Zero-Shot and Few-Shot Prowess: AbuseXLMR is remarkably data-efficient. Even with only 100 examples, it achieves performance levels that general models require much more data to reach. In zero-shot settings (training on one language and testing on another), it showed significantly higher transferability than its predecessors.
- Implicit Abuse Remains a Challenge: Error analysis revealed that models still struggle with "Implicit Abuse" (e.g., sarcastic comments about religious festivals) and highly subjective comments where annotators themselves experience confusion.
Limitations and Future Work
While MACD and AbuseXLMR represent a massive step forward, the authors acknowledge several hurdles. India has 22 official languages, and expanding coverage beyond the current five is a priority. Additionally, the dataset focuses primarily on native scripts, leaving a gap for "code-mixed" content where users write Indic languages using the Roman alphabet. Finally, because abuse is often subjective, future work aims to include real-valued scores to capture the intensity of comments rather than relying solely on binary labels.
Summary updated