SSNTrio@DravidianLangTech2025: LLM Based Techniques for Detection of Abusive Text Targeting Women

Mirnalinee T T, J Bhuvana, Avaneesh Koushik, Diya Seshan, Rohan RNAACL 2025

TL;DR

This paper tackles the critical issue of detecting abusive text targeting women on social media, specifically in the low-resource Dravidian languages of Tamil and Malayalam. By leveraging LLM-based techniques, the authors developed models that achieved competitive performance, highlighting the potential for language-specific models to outperform multilingual ones for nuanced language understanding.

What Problem Does This Paper Address?

Social media has become a double-edged sword. While it facilitates communication and information sharing, it also serves as a breeding ground for online abuse, with women disproportionately targeted by hateful and demeaning comments. Misogyny and harassment are rampant, causing significant emotional and psychological harm. Manually moderating the sheer volume of content is an insurmountable task, necessitating automated solutions. However, existing solutions often fall short when it comes to low-resource languages like Tamil and Malayalam, which are spoken by millions but lack extensive linguistic resources for NLP tasks. This paper addresses this gap by developing and evaluating LLM-based techniques for detecting abusive text specifically targeting women in these two languages.

Methods

The core of this work involves utilizing Large Language Models (LLMs) for text classification. The authors experimented with both multilingual and language-specific BERT models. The process began with data pre-processing, which included cleaning the text by removing special symbols and unnecessary punctuation while intentionally avoiding stopword removal to preserve crucial linguistic context. Tokenization was performed using pre-trained tokenizers for Tamil BERT and Malayalam MBERT.

The models were trained on a dataset of YouTube comments labeled as either "abusive" or "non-abusive." A significant portion of the methodology involved comparing the performance of a multilingual model (Multilingual-BERT or mBERT) against language-specific BERT models for Tamil and Malayalam. To combat overfitting, a common issue with LLMs, the researchers employed early stopping and explored weight decay.

Datasets

The study utilized a dataset of YouTube comments specifically curated for the "Abusive Tamil and Malayalam Text targeting Women on Social Media" shared task at DravidianLangTech@NAACL 2025. This dataset contains comments in Tamil and Malayalam, categorized into "abusive" and "non-abusive" classes. The paper indicates that the distribution of data points across these two classes is balanced for both languages, as visualized in Figures 1 and 2.

Key Insights and Ideas

  • Language-Specific Models Shine: The most striking insight is that language-specific BERT models (Tamil BERT and Malayalam BERT) significantly outperformed the multilingual BERT model (mBERT) for this task. This suggests that for nuanced tasks involving cultural context, idiomatic expressions, and subtle linguistic cues, fine-tuning on a single language's data allows models to capture these intricacies more effectively than a general-purpose multilingual model.
  • Overfitting is a Real Challenge: The paper explicitly notes that multilingual models exhibited overfitting, with training errors being much lower than testing errors. This underscores the importance of careful regularization techniques and model selection for low-resource languages.
  • Stopwords Matter for Context: The deliberate decision to not remove stopwords is a key methodological choice. This highlights the understanding that in certain contexts, especially in colloquial language or when identifying subtle abusive patterns, stopwords can carry significant semantic weight and contribute to contextual understanding.

Key Results

The study evaluated the performance of various models, including SVM, Logistic Regression, and different BERT variants. The final results, presented in Table 3, focus on the Macro F1-Scores for the language-specific BERT models:

  • Tamil BERT: Achieved a Macro F1-Score of 0.76, securing the 11th position in its subtask.
  • Malayalam BERT: Achieved a Macro F1-Score of 0.30, ranking 33rd in its subtask.

While the Malayalam score is lower, it's important to note the inherent challenges of low-resource languages and the significant improvement over what might be expected without dedicated models. The paper also presents training accuracies for other models in Tables 1 and 2, showing that while mBERT had a relatively high training accuracy (0.73 for Tamil, 0.69 for Malayalam), its performance on the test set (indicated by the Macro F1-Score) was likely impacted by overfitting.

Limitations and Future Work

The authors acknowledge several limitations and propose future directions:

  • Scalability to Other Languages: The current methodology is limited to languages for which dedicated pre-trained BERT models exist. For languages lacking such models due to insufficient training data, exploring multilingual models becomes a necessary alternative.
  • Improving Malayalam Performance: The lower performance of the Malayalam BERT model is attributed to potential overfitting on training data and the need for additional layers to learn task-specific patterns from BERT embeddings.
  • Advanced Techniques: Future work could involve applying similar methodologies to other regional languages. Advanced fine-tuning techniques like adversarial training, semi-supervised learning, or few-shot learning are suggested to enhance model adaptability. Data synthesis or text augmentation could also improve language coverage for models like the Malayalam BERT. Furthermore, exploring advanced tokenization techniques could help models handle complex linguistic structures more efficiently.

Summary updated

BibTeX
@inproceedings{t2025ssntrio,
  author = {Mirnalinee T T and J Bhuvana and Avaneesh Koushik and Diya Seshan and Rohan R},
  title = {{SSNTrio@DravidianLangTech2025: LLM Based Techniques for Detection of Abusive Text Targeting Women}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure