TL;DR
This paper tackles the critical issue of identifying abusive language targeting women on social media, specifically in Tamil and Malayalam. The researchers fine-tuned and compared three transformer models (BERT, XLM-ROBERTa, and DistilBERT) on a specialized dataset, finding XLM-ROBERTa to be the most effective, highlighting the challenges and importance of low-resource language NLP for creating safer online spaces.
What Problem Does This Paper Address?
The digital age has brought about a double-edged sword: while fostering communication, it also amplifies systemic gender-based violence through abusive online content. A staggering 40% of women worldwide report experiencing online harassment, with South Asian contexts being particularly vulnerable due to linguistic complexity and patriarchal norms. This abuse can manifest as threats, misogyny, and damaging stereotypes, leading to severe psychological distress and even physical danger. The challenge is further compounded by the scarcity of NLP resources for low-resource Dravidian languages like Tamil and Malayalam, which are spoken by millions but receive minimal AI research attention. This paper aims to address this gap by developing effective language-specific detection frameworks to combat "digital silencing" and promote safer online environments for women.
Methods
The study employs three prominent transformer-based models for text classification: BERT, XLM-ROBERTa, and DistilBERT. These models were fine-tuned on a dataset specifically curated for identifying abusive language in Tamil and Malayalam social media content. The researchers utilized AdamW for fine-tuning optimization with a learning rate of 2e-5. For pre-training, BERT and XLM-ROBERTa underwent two epochs, while DistilBERT, being a smaller model, was trained for three epochs to leverage its quicker training cycles. The performance of these models was evaluated using standard metrics like precision, recall, and F1-score.
Datasets
The primary dataset used in this study was provided by the DravidianLangTech 2025 shared task. This dataset consists of YouTube comments labeled as either "Abusive" or "Non-Abusive." For the development set, there were 2,790 instances for Tamil (1,366 abusive, 1,424 non-abusive) and 2,935 instances for Malayalam (1,531 abusive, 1,402 non-abusive). The test set comprised 598 instances for Tamil and 629 for Malayalam. The paper notes that the data distribution was generally balanced.
Key Insights and Ideas
- XLM-ROBERTa's multilingual prowess: The study strongly suggests that XLM-ROBERTa's superior performance, particularly in Tamil, stems from its extensive multilingual pre-training. This allows it to better identify cross-lingual similarities and transfer knowledge, which is crucial for languages with complex morphology.
- Low-resource language challenges are real: The significant performance drop observed for DistilBERT on Malayalam highlights the inherent difficulties in working with low-resource languages. This could be due to linguistic complexity, under-representation in pre-training data, or issues with the quality of the available data.
- Model choice is critical: The paper emphasizes that selecting the right model is paramount for effective abusive language detection, especially in linguistically diverse contexts.
- Beyond basic detection: The authors point towards future work involving explainable AI (XAI) methods like LIME and SHAP to understand why models make certain classifications, which can lead to more robust and interpretable systems.
Key Results
On the test set, XLM-ROBERTa achieved the highest F1-scores: 0.7708 for Tamil and 0.6876 for Malayalam. BERT followed with scores of 0.7658 (Tamil) and 0.6671 (Malayalam). DistilBERT showed more varied performance, with scores of 0.7639 (Tamil) and a significantly lower 0.4876 (Malayalam). The paper notes that while models were consistent for Tamil, Malayalam presented a greater challenge, with DistilBERT exhibiting near-random performance. The team Hydrangea achieved 9th and 12th ranks with macro F1 scores of 0.7708 and 0.6769 for Tamil and Malayalam, respectively, on the leader board.
Limitations and Future Work
The authors acknowledge that the transformer models used in their experiments struggled with identifying patterns in code-mixed Dravidian languages, leading to lower F1 scores. This limitation is attributed to the dataset size not being adequate for training the transformer models. They suggest that data augmentation techniques could be applied in the future to overcome this.
Future work directions include:
- Utilizing language-agnostic embeddings to enhance performance.
- Applying Explainable AI (XAI) methods (LIME, SHAP) to understand model decision-making.
- Employing attention visualization to identify input text aspects the model focuses on.
- Exploring zero-shot or few-shot learning methods to mitigate data sparsity by leveraging knowledge from other languages or tasks.
Summary updated