TL;DR
This paper tackles the critical challenge of detecting and categorizing abusive language in Tamil, a low-resource language. The authors developed a system, COMBATANT, that uses Logistic Regression with TF-IDF features to achieve strong performance, securing second place in the shared task. Their work highlights the effectiveness of traditional ML models even in the face of advanced deep learning approaches for this specific task.
What Problem Does This Paper Address?
The proliferation of social media has led to a significant increase in abusive language online, posing risks like cyberbullying and harassment. While considerable research has focused on detecting such content in high-resource languages, low-resource languages like Tamil have been largely overlooked. Manually moderating the vast amount of online content is infeasible, necessitating the development of intelligent automated systems. This paper aims to address this gap by building a system capable of fine-grained categorization of abusive comments in Tamil.
Methods
The authors explored a range of machine learning (ML), deep learning (DL), and transformer-based models for abusive comment detection. Their approach involved:
- Feature Extraction: TF-IDF for unigrams, Word2Vec, and FastText embeddings were used.
- ML Baselines: Logistic Regression (LR), Decision Tree (DT), and Support Vector Machine (SVM) were implemented. A majority voting ensemble was also explored.
- DL Baselines: A CNN-BiLSTM model was constructed, incorporating both Word2Vec and FastText embeddings.
- Transformer-based Model: Indic-BERT, a pre-trained model for Indian languages, was fine-tuned.
The core of their winning approach, COMBATANT, relied on Logistic Regression with TF-IDF features.
Datasets
The study utilized a gold standard dataset created by the task organizers for detecting abusive comments in Tamil social networking sites. This dataset was divided into training, validation, and test sets. The dataset is characterized by an imbalance in class distribution, with some categories like Transphobia and Homophobia having very few samples. The task required classifying texts into eight predefined categories: Misogyny, Misandry, Homophobia, Transphobia, Xenophobia, Counter-speech, Hope-speech, and None-of-the-above.
Key Insights and Ideas
- Logistic Regression's Resilience: Despite the prevalence of advanced DL and transformer models, Logistic Regression with TF-IDF features emerged as the top-performing model for this specific Tamil abusive language detection task. This suggests that simpler, well-established methods can still be highly effective, especially when dealing with specific linguistic nuances or data characteristics.
- The Power of TF-IDF: For this particular dataset and task, TF-IDF proved to be a more effective feature representation than Word2Vec or FastText embeddings when used with Logistic Regression. This highlights that the choice of feature extraction is as crucial as the choice of model.
- Transformer Model Underperformance: Indic-BERT, a promising transformer model for Indian languages, did not perform as well as expected. The authors hypothesize this could be due to the model's unfamiliarity with local Tamil words or the prevalence of local slang within the dataset.
- Class Imbalance Challenges: The significant class imbalance in the dataset, particularly for categories like Transphobia, posed a challenge and likely contributed to lower performance in those specific classes.
Key Results
The COMBATANT system, employing Logistic Regression with TF-IDF features, achieved the highest macro F₁-score of 0.39 among the ML models. This performance secured them the 2nd rank in the shared task competition. While the CNN+BiLSTM model with FastText embeddings also achieved a macro F₁-score of 0.39, the Logistic Regression model demonstrated a higher recall value (0.44 vs. 0.36). The Indic-BERT transformer model performed poorly, achieving a macro F₁-score of 0.19.
Limitations and Future Work
The authors acknowledge the class imbalance in the dataset as a significant limitation, which likely impacted the performance of certain categories. They suggest that adding more diverse data could help address this issue. Future work could involve exploring more advanced transformer models like XLM-R, Electra, mBERT, or MuRIL to see if they can better handle the nuances of Tamil text and overcome the limitations observed with Indic-BERT. Investigating techniques to better handle class imbalance is also a promising direction.
Summary updated