M-BAD: A Multilabel Dataset for Detecting Aggressive Texts and Their Targets

Omar Sharif, Eftekhar Hossain, Mohammed Moshiul HoqueACL 2022

TL;DR

This paper introduces M-BAD, a novel Bengali dataset for detecting aggressive texts and their targets, addressing a critical gap in low-resource languages. They developed a two-level annotation scheme and evaluated various ML, DL, and transformer models, finding Bangla-BERT to be the most effective, though challenges remain in identifying context-dependent aggression.

What Problem Does This Paper Address?

The proliferation of aggressive, abusive, and hateful content on social media poses a significant societal problem. While research has focused on high-resource languages like English, low-resource languages, such as Bengali, have been largely overlooked. This is problematic because users increasingly communicate in their regional languages online, and models trained on English often fail to detect harmful content in Bengali due to linguistic differences. Existing work also often treats aggression targets as mutually exclusive, whereas a single text can target multiple categories simultaneously. This paper aims to bridge this gap by creating a dedicated resource and benchmark for Bengali aggressive text detection with multilabel target identification.

Methods

The authors employed a two-level hierarchical annotation schema for their dataset. Level 1 categorizes texts as either aggressive (AG) or non-aggressive (NoAG). Level 2 further classifies aggressive texts into five fine-grained target classes: religious aggression (ReAG), political aggression (PoAG), verbal aggression (VeAG), gendered aggression (GeAG), and racial aggression (RaAG). They considered overlapping target classes, a departure from previous work.

For model evaluation, they explored several approaches:

  • ML-based methods: Logistic Regression (LR) and Naive Bayes with Support Vector Machine (NBSVM), using Bag-of-Words features.
  • DL-based methods: BiGRU combined with FastText embeddings.
  • Pretrained Transformers: Specifically, multilingual BERT (m-BERT), BERT for Bangla (Bangla-BERT), and BERT for Indian languages (Indic-BERT). These models were fine-tuned on the M-BAD dataset.

Datasets

The primary contribution is the introduction of M-BAD, a new multilabel Bengali dataset containing 15,650 texts. This dataset was manually collected from YouTube and Facebook posts and comments between June and December 2021, focusing on content posted after January 1, 2020. The dataset is split into training (80%), testing (10%), and validation (10%) sets using a stratified approach. The authors also reference existing datasets for non-Bengali aggressive text classification, such as those compiled by Kumar et al. and Bhardwaj et al., and for Bengali by Chakraborty and Seddiqui, and Sharif and Hoque.

Key Insights and Ideas

  • Low-resource languages are critical: The paper highlights the urgent need for resources and models for languages like Bengali, where online aggression can have real-world consequences.
  • Multilabel aggression is complex: Recognizing that aggression can target multiple categories simultaneously (e.g., political and gendered) is crucial for a more nuanced understanding.
  • Transformer models excel, but context is king: While transformer models like Bangla-BERT show superior performance, identifying subtle, context-dependent, sarcastic, or implicit aggression remains a significant challenge.
  • Data imbalance is a hurdle: The fine-grained target classes in M-BAD are imbalanced, posing difficulties for model training and evaluation.

Key Results

  • Coarse-grained classification: Bangla-BERT achieved the highest weighted F1-score of 0.92 for distinguishing aggressive from non-aggressive texts.
  • Multilabel fine-grained classification: Bangla-BERT also led in this task, achieving a weighted F1-score of 0.83 and a macro F1-score of 0.72.
  • Model performance: Transformer models generally outperformed ML and BiGRU models.
  • Annotation agreement: Inter-annotator agreement, measured by Kappa scores, was higher for the coarse-grained classification (0.77) than for the fine-grained targets (around 0.62), indicating greater difficulty in identifying specific aggression targets.
  • Error analysis: The model struggled most with the racial aggression (RaAG) class, likely due to its small sample size. Implicit and ambiguous language also contributed to misclassifications.

Limitations and Future Work

The authors acknowledge that the dataset's class distribution is imbalanced, particularly for the fine-grained targets, which presents a challenge. They also note the difficulty in identifying context-dependent aggression, often expressed through sarcasm or implicit language. Future work suggested includes:

  • Developing multitask learning approaches to better handle overlapping aggression targets.
  • Exploring domain adaptation techniques.
  • Expanding the dataset with more samples covering a wider time range to mitigate bias towards specific events.
  • Addressing the class imbalance issue by collecting more data for under-represented categories.

Summary updated

BibTeX
@inproceedings{sharif2022m,
  author = {Omar Sharif and Eftekhar Hossain and Mohammed Moshiul Hoque},
  title = {{M-BAD: A Multilabel Dataset for Detecting Aggressive Texts and Their Targets}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure