TL;DR
This paper presents the findings of a shared task focused on detecting abusive text targeting women in Tamil and Malayalam, two low-resource Dravidian languages. The study highlights the effectiveness of transformer-based models and identifies key challenges and promising directions for future research in this critical area of online safety.
What Problem Does This Paper Address?
Gender-based violence (GBV) is a pervasive issue, and online spaces are no exception. While significant progress has been made in detecting abusive content targeting women in high-resource languages like English, there's a substantial gap in resources and models for low-resource languages, particularly Dravidian languages such as Tamil and Malayalam. This lack of tools leaves women speaking these languages more vulnerable to online harassment. This paper addresses this gap by organizing a shared task to encourage the development of robust systems for identifying abusive text specifically targeting women in these languages, using YouTube comments as the data source.
Methods
The shared task involved participants developing models to classify YouTube comments in Tamil and Malayalam as either "Abusive" or "Non-Abusive." The core methodology revolved around leveraging natural language processing (NLP) techniques. A significant trend observed among participants was the extensive use of transformer-based models, including popular architectures like BERT, mBERT, MuRIL, and XLM-ROBERTa. Many teams fine-tuned these pre-trained models on the provided annotated datasets. Some participants also explored hybrid approaches, combining transformer embeddings with traditional machine learning classifiers like Random Forest, SVM, and Logistic Regression. Techniques like data augmentation and hyperparameter tuning were employed to address challenges such as class imbalance.
Datasets
The study utilized a newly curated dataset of YouTube comments in Tamil and Malayalam. These comments were scraped from YouTube using targeted queries focused on controversial topics where abuse against women is commonly found. The dataset was annotated by six annotators (four computer science students and two social work students) to label comments as either "Abusive" or "Non-Abusive." The paper reports on the Inter-Annotator Agreement (IAA) for both languages, with Krippendorff's Alpha values indicating moderate agreement for Tamil (0.6474) and near-perfect agreement for Malayalam (0.9573). The dataset was split into training, validation, and testing sets for the shared task.
Key Insights and Ideas
- Transformer Dominance: The overwhelming success and widespread adoption of transformer-based models (BERT, XLM-ROBERTa, etc.) underscore their effectiveness in capturing the nuances of language, even in low-resource settings.
- Low-Resource Language Challenges: Detecting abusive content in Tamil and Malayalam presents unique difficulties due to their agglutinative nature, frequent code-switching, and the inherent scarcity of large annotated datasets.
- Importance of Annotation Quality: The high IAA for Malayalam suggests that careful annotation guidelines and diverse annotator backgrounds contribute to reliable datasets, which are crucial for training effective models.
- Targeted Abuse Detection: The task's focus on women-targeted abuse is a critical distinction from general hate speech detection, highlighting the need for models that can understand gender-specific nuances and harmful language patterns.
Key Results
The shared task was evaluated using the macro-averaged F1-score.
- Tamil: The top-performing team, CUET_Agile, achieved a macro F1-score of 0.7883. Other notable performers included MSM_CUET (0.7873) and Incepto (0.7864). The results demonstrate that fine-tuned transformer models, particularly those leveraging optimization strategies, were highly effective.
- Malayalam: The top-performing team, Habiba A, G Agila, achieved a macro F1-score of 0.7571, utilizing a Recurrent Neural Network (RNN) approach combined with effective preprocessing. CUET_Agile (0.7234) and CUET_Novice (0.7083) secured the second and third positions, respectively, using fine-tuned transformer models.
Overall, the results indicate that transformer-based models generally outperformed traditional approaches, with models like mBERT and XLM-ROBERTa showing strong performance.
Limitations and Future Work
The authors acknowledge that the current work focuses on binary classification (Abusive/Non-Abusive). Future work could expand this to multiclass problems, including stereotype detection, bias detection, and gender-neutral term analysis. This would enable a more nuanced and contextual understanding of abusive content. The paper also suggests that further research into improving preprocessing techniques and exploring more advanced model architectures tailored for Dravidian languages could yield even better results. The development of more comprehensive datasets and shared tasks will be crucial for advancing research in this vital area.
Summary updated