TL;DR
This paper tackles the challenge of detecting abusive comments in Tamil, a low-resource language, by fine-tuning transformer models like XLM-RoBERTa and DeBERTa. They achieved competitive results on a novel dataset, highlighting the effectiveness of these models for under-resourced languages and paving the way for better online safety tools.
What Problem Does This Paper Address?
The proliferation of social media has unfortunately led to an increase in negative and abusive online content. While significant research has focused on detecting such content in high-resource languages like English, there's a considerable gap for low-resource languages, including Tamil. This lack of research and readily available datasets hinders the development of effective tools to combat online abuse in these linguistic communities. This paper aims to address this by applying state-of-the-art transformer models to the task of abusive comment classification in Tamil, both in pure Tamil and in code-mixed Tamil-English.
Methods
The core methodology involves fine-tuning two powerful pre-trained transformer models: XLM-RoBERTa and DeBERTa. These models, known for their strong performance in Natural Language Processing (NLP) tasks, were adapted to the specific problem of abusive comment classification. The authors chose not to preprocess the text by removing stop words, special characters, or emojis, believing these elements provide crucial context for understanding the nuances of online communication. They trained separate models for two sub-tasks: one on pure Tamil text and another on code-mixed Tamil-English text.
Datasets
The research utilizes a novel dataset created for the "Abusive Comment Detection" shared task at the Second Workshop on Speech and Language Technologies for Dravidian Languages (ACL 2022). This dataset consists of YouTube comments annotated with eight classes of abusive comments. The dataset is divided into two sub-tasks:
- Sub-task A: Pure Tamil text.
- Sub-task B: Code-mixed Tamil-English text.
Both datasets contain the same eight abusive comment categories. The paper provides details on the number of rows in the training, development, and test splits for each sub-task.
Key Insights and Ideas
- Transformer models are effective for low-resource languages: The study demonstrates that even without extensive language-specific pre-training, transformer models like XLM-RoBERTa can achieve good performance on Tamil text.
- Context is king (even for emojis and special characters): The decision to retain special characters and emojis in the text proved beneficial, as these elements often carry significant contextual information in online comments.
- Model strengths vary by language mix: XLM-RoBERTa, being a multilingual model, showed better performance on the pure Tamil sub-task (A), while DeBERTa, which has strong English pre-training, excelled on the code-mixed Tamil-English sub-task (B). This highlights the importance of considering the language characteristics when selecting a model.
- Code-mixing presents unique challenges and opportunities: The higher performance on the code-mixed dataset suggests that the English components within the text might have provided additional signals that the models could leverage, given their English pre-training.
Key Results
The paper reports the following classification metrics:
- Sub-task A (Tamil text):
- XLM-RoBERTa achieved an accuracy of 0.66, with a weighted F1-score of 0.65, precision of 0.65, and recall of 0.66.
- DeBERTa achieved an accuracy of 0.62, with a weighted F1-score of 0.57, precision of 0.62, and recall of 0.56.
- Sub-task B (Code-mixed Tamil-English text):
- XLM-RoBERTa achieved an accuracy of 0.72, with a weighted F1-score of 0.70, precision of 0.70, and recall of 0.72.
- DeBERTa achieved an accuracy of 0.72, with a weighted F1-score of 0.72, precision of 0.72, and recall of 0.72.
The authors' team ranked 2nd in the code-mixed classification sub-task and 8th in the Tamil-text sub-task, indicating strong performance in a competitive setting.
Limitations and Future Work
The authors acknowledge several areas for future work:
- Handling "Not Tamil" instances: For sub-task A, the presence of a few "Not Tamil" instances in the training data, though under-sampled, could potentially be removed to improve performance.
- Leveraging emoji information: Further research could explore extracting emoji information separately and incorporating it more explicitly into the classification process.
- Custom model architectures: While pre-trained models are a good starting point, the authors suggest that developing custom model architectures specifically tailored for Tamil text classification could lead to even better results.
- Addressing class imbalance: The paper mentions that some classes in the datasets are under-sampled, which is a common challenge in text classification that could be further investigated.
Summary updated