TL;DR
This paper tackles the crucial challenge of detecting abusive comments in Tamil, a low-resource language, using two distinct deep learning models: an n-gram-MLP and a 1D Conv-LSTM. The n-gram-MLP model emerged as the stronger performer, highlighting the effectiveness of character n-grams for this task, even in code-mixed Tamil-English contexts.
What Problem Does This Paper Address?
Abusive language, encompassing hate speech, profanity, and cyberbullying, is a pervasive issue on online platforms, causing societal harm and complicating moderation efforts. While significant progress has been made in detecting such content in high-resource languages like English, low-resource languages, particularly Dravidian languages like Tamil, remain largely unexplored. This is primarily due to the scarcity of annotated datasets and the inherent complexities of these languages, including code-mixing (blending Tamil with English) and the use of informal language, slang, and abbreviations. The "Abusive Comment Detection in Tamil-ACL 2022" shared task aims to bridge this gap by encouraging the development of models for detecting abusive content in both native Tamil script and code-mixed Tamil texts.
Methods
The authors from team MUCIC submitted two models to the shared task:
- n-gram-MLP Model: This model utilizes character n-grams (sequences of 1 to 3 characters) extracted from the text. These n-grams are then vectorized using TF-IDF (Term Frequency-Inverse Document Frequency) and fed into a Multilayer Perceptron (MLP) classifier. The MLP was configured with specific hidden layer sizes, a maximum number of iterations, and an Adam optimizer.
- 1D Conv-LSTM Model: This model employs a more complex deep learning architecture. It begins with tokenizing the text and then uses "pad_sequences" to ensure uniform sequence lengths. A 1D convolutional layer with ReLU activation is followed by pooling layers. The core of the model consists of fully connected LSTM (Long Short-Term Memory) layers, culminating in a soft-max output layer for classification.
Both models first undergo a text cleaning process to remove punctuation, numerical data, frequent words, and stop-words, aiming to improve model performance.
Datasets
The primary datasets used in this study were provided by the "Abusive Comment Detection in Tamil-ACL 2022" shared task. These datasets consist of:
- Native Tamil Script Texts: Comments written solely in the Tamil language.
- Code-Mixed Tamil Texts (Ta-En): Comments that blend Tamil and English.
The datasets are divided into training and testing sets. The paper notes that these datasets are imbalanced, with a significant disparity in the number of samples across different abusive categories, which poses a challenge for classification.
Key Insights and Ideas
- Character n-grams are surprisingly effective: Despite the complexity of Tamil and code-mixed texts, simple character n-grams, when combined with an MLP, proved to be a strong baseline and even outperformed the more sophisticated 1D Conv-LSTM model. This suggests that local character patterns are highly indicative of abusive content in this context.
- Code-mixing adds significant complexity: The performance difference between native Tamil and code-mixed Tamil texts, even for the best-performing model, highlights the challenges introduced by language mixing.
- Low-resource language challenges persist: The paper reiterates the well-known difficulties in low-resource NLP, particularly the scarcity of annotated data and the inherent linguistic complexities that make automatic detection challenging.
Key Results
The n-gram-MLP model achieved the best performance for team MUCIC:
- Weighted F1-score of 0.560 for code-mixed Tamil texts.
- Weighted F1-score of 0.430 for native Tamil script texts.
While the 1D Conv-LSTM model was also submitted, it did not perform as well as the n-gram-MLP. The paper also presents results on a development set, showing similar trends. The n-gram-MLP model secured a respectable position among the participating teams in the shared task.
Limitations and Future Work
- Dataset Imbalance: The authors acknowledge the imbalanced nature of the provided datasets as a significant challenge that complicates the classification task.
- Performance Gap: The performance gap between native Tamil and code-mixed Tamil texts indicates room for improvement in handling code-mixed language.
- Future Work: While not explicitly detailed, the paper implies that further research could focus on more robust methods for handling code-mixed languages, potentially exploring advanced feature engineering or specialized deep learning architectures. Investigating techniques to mitigate the impact of data imbalance would also be a valuable direction.
Summary updated