TL;DR
This paper tackles the challenging task of detecting abusive comments in Tamil-English code-mixed text, a common phenomenon on social media. The authors propose a novel approach by combining language-agnostic sentence embeddings from LaBSE with TF-IDF features and a curated vocabulary, which they then feed into an SVM classifier. This method achieved a macro F1-score of 0.54 on their custom dataset, demonstrating a promising direction for handling low-resource, code-mixed languages.
What Problem Does This Paper Address?
Online platforms are rife with abusive comments, which can have detrimental effects on users' mental well-being. Identifying and moderating this content is crucial, but it becomes significantly harder when dealing with low-resource languages like Tamil, especially when combined with English (code-mixing). Most existing tools and pre-trained models are heavily biased towards English, leaving a gap in effectively handling such multilingual online discourse. This paper aims to bridge this gap by developing a system specifically for Tamil-English code-mixed abusive comment detection.
Methods
The core of the proposed methodology lies in creating effective text representations (embeddings) for the code-mixed data. The authors explore three main approaches:
- TF-IDF Vectorization: A traditional method that quantifies word importance within a document. For this paper, they created a custom vocabulary by selecting the top 100 most frequent words from each category to focus on statistically relevant terms for abusive comments.
- LaBSE (Language-Agnostic BERT Sentence Embedding): A state-of-the-art model that encodes sentences into a shared embedding space, making it language-agnostic. This is particularly beneficial for low-resource languages as it can leverage knowledge from other languages. LaBSE embeddings were generated with a dimension of 700.
- Custom Embeddings: This novel approach combines the strengths of both TF-IDF and LaBSE. They first generate individual embeddings using both methods and then concatenate them to create a richer, custom representation.
These embeddings are then fed into various classifiers, including SVM, MLP, and K-Nearest Neighbors. The paper highlights that SVM, particularly with an RBF kernel, generally performed best.
Datasets
The study utilizes a custom dataset of comments collected from YouTube videos related to Tamil content. These comments were manually annotated into eight categories: transphobia, counter-speech, misandry, homophobia, hope-speech, xenophobia, misogyny, and none-of-the-above. A significant characteristic of this dataset is the disproportionate representation of the "none-of-the-above" category, with 62% in the training set and 61% in the development set.
Key Insights and Ideas
- The Power of Combination: The most significant insight is that combining language-agnostic embeddings (LaBSE) with traditional TF-IDF features, especially when using a curated vocabulary, leads to better performance than using either method in isolation for code-mixed abusive comment detection.
- LaBSE's Advantage for Low-Resource Languages: The paper reinforces the utility of LaBSE for languages with limited resources, as it can effectively capture semantic meaning across language barriers.
- SVM's Robustness: The Support Vector Machine (SVM) classifier, particularly with an RBF kernel, consistently emerged as a strong performer across different embedding strategies.
- Challenges of Code-Mixing: The inherent difficulty in processing code-mixed text, especially with variations in transliteration and spelling, is a recurring theme, underscoring the need for specialized approaches.
Key Results
The proposed model, which combines TF-IDF and LaBSE embeddings with an SVM classifier, achieved:
- An accuracy of 52% and a macro F1-score of 0.54 on the development dataset.
- On the test dataset, the model achieved a macro F1-score of 0.34, a precision score of 0.33, and a recall score of 0.37.
- The paper also reports that the SVM classifier with custom embeddings achieved the best performance with an accuracy of 0.74 on the development dataset.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Dataset Imbalance: The significant imbalance in the dataset, particularly the large "none-of-the-above" category, poses a challenge.
- Preprocessing Limitations: The current preprocessing steps involve removing special characters and emoticons. The authors suggest that acknowledging and utilizing these elements could potentially improve performance, as they often carry significant sentiment or context in online communication.
- Exploring Better Preprocessing: Future work could focus on more sophisticated preprocessing techniques to better handle the nuances of code-mixed text.
- Investigating Other Models: While SVM performed well, exploring other advanced models or ensemble techniques could yield further improvements.
Summary updated