MSM_CUET@DravidianLangTech 2025: XLM-BERT and MuRIL Based Transformer Models for Detection of Abusive Tamil and Malayalam Text Targeting Women on Social Media

Md Mizanur Rahman, Srijita Dhar, Md Mehedi Hasan, Hasan MuradNAACL 2025

TL;DR

This paper tackles the critical challenge of detecting abusive text targeting women on social media, specifically for low-resource Dravidian languages like Tamil and Malayalam. By fine-tuning powerful transformer models like XLM-R and MuRIL, the researchers achieved state-of-the-art results on a specialized shared task dataset, highlighting the effectiveness of these models for nuanced language understanding.

What Problem Does This Paper Address?

The proliferation of social media has unfortunately led to an increase in online harassment and abusive language. While detecting such content is crucial, it becomes significantly more challenging for low-resource languages where data and pre-trained models are scarce. This paper focuses on the specific problem of identifying abusive text that targets women on social media in Tamil and Malayalam, two prominent languages in South India. The cultural and linguistic intricacies of these languages pose unique hurdles for existing Natural Language Processing (NLP) models, which often struggle to capture the subtle nuances of abusive language in these contexts.

Methods

The researchers employed a range of transformer-based models, including XLM-RoBERTa (XLM-R), MuRIL, IndicBERT, and mBERT. They also explored ensemble methods combining these transformers with traditional machine learning classifiers like Support Vector Machines (SVM) and Random Forests. The core of their approach involved fine-tuning these pre-trained multilingual models on a specialized dataset for abusive text detection. They specifically selected XLM-R for Tamil and MuRIL for Malayalam due to their superior performance during initial experiments. The models were trained using an 80%-20% train-validation split, with mixed precision training (fp16) utilized for efficiency. Early stopping was implemented to prevent overfitting by monitoring validation loss.

Datasets

The primary dataset used in this study is from the DravidianLangTech@NAACL 2025 shared task. This dataset contains abusive and non-abusive text samples for both Tamil and Malayalam languages. It was divided into training, development, and test sets. Notably, the test set was unlabeled, requiring the models to predict the labels. The authors also mention standardizing labels that had inconsistencies (e.g., 'abusive' vs. 'Abusive') to 'Abusive' for uniformity.

Key Insights and Ideas

  • Transformer Dominance for Low-Resource Languages: The study strongly suggests that fine-tuned multilingual transformer models like XLM-R and MuRIL are highly effective for abusive text detection in low-resource languages, outperforming traditional methods and even some other transformer architectures.
  • Language-Specific Model Choice Matters: The research highlights that the optimal transformer model can vary between languages. XLM-R proved superior for Tamil, while MuRIL excelled for Malayalam, underscoring the importance of language-specific evaluation.
  • Ensemble Methods Not Always Superior: Surprisingly, the ensemble methods, while often powerful, did not yield the best results in this specific task, suggesting that a well-tuned single transformer model can be more effective than combining multiple models.
  • Challenges of Nuance: The authors acknowledge that even advanced models struggle with capturing sarcasm, irony, and implicit abusive content due to the inherent linguistic complexities and limited data for these low-resource languages.

Key Results

For abusive Tamil text detection on the test set, the XLM-R model achieved the highest F1 score of 0.7873, ranking 2nd among all participants in the shared task. For abusive Malayalam text detection, the MuRIL model achieved the highest F1 score of 0.6812, ranking 10th. On the validation set, XLM-R achieved 79% accuracy and an F1 score of 0.79 for Tamil, while MuRIL achieved 73% accuracy and an F1 score of 0.73 for Malayalam.

Limitations and Future Work

The authors acknowledge two primary limitations:

  1. Small Dataset Size: The dataset provided for the shared task was relatively small, which they believe impacted model development and performance.
  2. Capturing Nuance: Their models showed limitations in capturing sarcasm, irony, and implicit abusive content. This is attributed to the low-resource nature of the languages and their native morphology, making context challenging to interpret.

Future work could involve incorporating additional context through hierarchical models or fine-tuning multilingual transformers on domain-specific corpora to improve performance. Exploring methods to better handle sarcasm and irony in these languages would also be a valuable direction.

Summary updated

BibTeX
@inproceedings{rahman2025msm,
  author = {Md Mizanur Rahman and Srijita Dhar and Md Mehedi Hasan and Hasan Murad},
  title = {{MSM\_CUET@DravidianLangTech 2025: XLM-BERT and MuRIL Based Transformer Models for Detection of Abusive Tamil and Malayalam Text Targeting Women on Social Media}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure