TL;DR
This paper tackles the challenging problem of detecting gender-targeted abusive language in Tamil and Malayalam social media, which is complicated by code-mixing and linguistic nuances. The authors evaluate traditional machine learning models (SVM, LR, RFC) and find SVM to be the most effective, though they highlight the need for more advanced, context-aware models for subtle abuse.
What Problem Does This Paper Address?
The proliferation of social media has unfortunately led to an increase in online abuse, particularly gender-based harassment targeting women. While abusive language detection has seen progress in high-resource languages like English, it remains a significant challenge for low-resource languages, especially Dravidian languages like Tamil and Malayalam. These languages present unique difficulties due to their complex syntax, rich morphology, code-mixing (interspersing words from different languages, e.g., Tamil and English), and culturally specific informal expressions. Existing methods often struggle with these nuances, leading to under-detection of subtle or implicit abuse. This paper aims to address this gap by evaluating the effectiveness of standard machine learning models for this specific task.
Methods
The researchers employed three traditional machine learning models:
- Support Vector Machine (SVM): Chosen for its ability to find precise classification boundaries.
- Logistic Regression (LR): A linear model selected for its interpretability and effectiveness in binary classification.
- Random Forest Classifier (RFC): An ensemble method that uses multiple decision trees to improve predictive performance.
For feature extraction, the TF-IDF (Term Frequency-Inverse Document Frequency) vectorization technique was used. This standard approach converts text into numerical features, weighting words based on their frequency in a document and their rarity across the corpus. Preprocessing involved cleaning the data, handling missing values, removing stop words, and mapping labels to binary values (0 for non-abusive, 1 for abusive). The models were trained and evaluated using standard classification metrics.
Datasets
The study utilized a custom dataset scraped from YouTube comments, specifically focusing on content related to controversial and sensitive topics to ensure a diverse representation of abusive and non-abusive text. The dataset was divided into:
- Training set: 5,723 texts
- Development set: 1,229 texts
- Test set: 1,227 texts
Each text was labeled as either "Abusive" or "Non-Abusive." The paper provides sample texts from both Tamil and Malayalam datasets to illustrate the linguistic challenges, including code-mixing and informal language.
Key Insights and Ideas
- SVM Outperforms Traditional Models: Despite the complexities of Tamil and Malayalam, SVM emerged as the most effective among the traditional machine learning models tested, showing superior performance in accuracy and recall.
- Code-Mixing is a Major Hurdle: The paper explicitly identifies code-mixed sentences (Tamil-Malayalam) as a significant challenge, where TF-IDF struggles to capture nuanced language patterns, leading to misclassifications.
- Implicit Abuse Remains Elusive: Detecting subtle, implicit, or culturally nuanced forms of abuse is still a major difficulty, even for the best-performing models. This suggests a need for deeper contextual understanding.
- Low-Resource Language Challenges: The scarcity of large, annotated datasets and the inherent linguistic richness of Tamil and Malayalam pose ongoing challenges for robust abuse detection.
Key Results
The study evaluated the performance of Logistic Regression, Random Forest, and SVM on both Tamil and Malayalam datasets.
For Tamil:
- SVM achieved the highest accuracy (69%) and recall (71%) for abusive texts.
- Overall, SVM showed slightly better performance across all metrics compared to LR and RFC.
For Malayalam:
- SVM also performed best, achieving an accuracy of 64% and recall of 62% for abusive texts.
- Performance was generally lower than for Tamil, likely reflecting the increased complexity or specific characteristics of the Malayalam data.
Confusion matrices for both datasets are provided, visually representing the model's predictions versus the true labels.
Limitations and Future Work
The authors acknowledge several limitations:
- Limited Context Understanding: Traditional models like SVM struggle with indirect and implicit abusive language due to their limited ability to understand context.
- Code-Mixing Challenges: TF-IDF's inability to effectively handle code-mixed sentences leads to false negatives.
- Scarcity of Labeled Datasets: The limited size of annotated datasets in these languages restricts model effectiveness.
- Intricate Grammatical Structures: The complex syntax and diverse vocabulary of Tamil and Malayalam complicate accurate identification.
- Cultural Nuances: Recognizing gender-based abuse requires a deeper understanding of cultural context, which current models lack.
Future work proposed includes:
- Exploring transformer-based models like BERT for better context understanding.
- Implementing data augmentation techniques to enhance model performance.
- Expanding labeled datasets to improve abuse detection in these low-resource languages.
- Investigating transliteration normalization and custom tokenization strategies to better handle transliterated words and mixed scripts.
- Addressing ethical considerations for unbiased predictions.
The dataset and implementation code are made publicly available to support reproducibility and further research.
Summary updated