DLRG@DravidianLangTech-ACL2022: Abusive Comment Detection in Tamil using Multilingual Transformer Models

Ratnavel Rajalakshmi, Ankita Duraphe, Antonette ShibaniACL 2022

TL;DR

This paper tackles the challenge of detecting abusive comments in Tamil, a language often mixed with English, by exploring various machine learning, deep learning, and transformer-based models. They found that while transformer models like mBERT show promise for pure Tamil, traditional machine learning approaches like Random Forest surprisingly outperformed them on mixed-language data, highlighting the complexities of code-mixed language processing.

What Problem Does This Paper Address?

The internet and social media have revolutionized communication, but they've also become breeding grounds for online abuse. Detecting this abusive content is a significant challenge, especially in languages like Tamil, where users frequently mix Tamil and English (code-mixing). This linguistic fluidity makes it difficult for automated systems to accurately identify and categorize harmful comments. Existing methods often struggle with the nuances of such code-mixed text, necessitating the development of specialized approaches. This paper aims to address this gap by building and evaluating systems for abusive comment detection in Tamil, considering both pure Tamil and Tamil-English code-mixed datasets.

Methods

The authors employed a multi-pronged approach, experimenting with three main categories of models:

  • Machine Learning (ML): They implemented eight traditional ML algorithms, including Logistic Regression, Random Forest, K-Nearest Neighbors, Decision Tree, Support Vector Machine, Gradient Boosting, Adaptive Boosting, and Ensemble methods. For feature extraction, they used TF-IDF.
  • Deep Learning (DL): Two Bi-directional LSTM (BiLSTM) models were used. BiLSTM-M1 incorporated convolution and max-pooling layers for feature extraction, while BiLSTM-M2 utilized pre-trained word embeddings.
  • Transformer-based Models: They fine-tuned two transformer models: mBERT (bert-base-multilingual-cased) and IndicBERT, which were pre-trained on a large number of languages, including Tamil.

To handle class imbalance, a common issue in such datasets, they focused on using the F1-score as the primary evaluation metric and considered penalized algorithms.

Datasets

The study utilized datasets provided by the DravidianLangTech-ACL 2022 competition. These datasets consist of YouTube comments annotated for various abusive categories, including Misogyny, Misandry, Homophobia, Transphobic, Xenophobia, Counter-Speech, and Hope-Speech, along with a "None-of-the-above" category. The authors worked with two versions:

  • Tamil+English Dataset: Comments written in Tamil and English, exhibiting code-mixing.
  • Tamil Dataset: Comments written purely in Tamil.

The paper provides tables detailing the number of comments and class distribution for both datasets across train, validation, and test splits.

Key Insights and Ideas

  • Code-Mixing is a Major Hurdle: The performance differences between the Tamil and Tamil+English datasets highlight how challenging code-mixed language is for NLP models.
  • Traditional ML Shines on Mixed Data: Surprisingly, Random Forest, a traditional ML algorithm, achieved the best results on the Tamil+English dataset, outperforming deep learning and transformer models. This suggests that simpler models might be better at capturing the patterns in code-mixed text when complex linguistic features are hard to disentangle.
  • Transformer Models Excel in Pure Tamil: For the pure Tamil dataset, transformer models, specifically mBERT, showed superior performance, indicating their strength in handling monolingual text with rich linguistic structures.
  • Pre-trained Embeddings Aren't Always Enough: The authors noted that pre-trained word embeddings didn't always lead to better performance in DL models, possibly due to a lack of feature mapping between words in code-mixed contexts.

Key Results

  • Tamil+English Dataset: Random Forest achieved the best weighted average F1-score of 0.78.
  • Tamil Dataset: mBERT achieved the best weighted average F1-score of 0.7.
  • Bi-Directional LSTM (BiLSTM-M2) also performed well on the Tamil dataset with a weighted average F1-score of 0.70.

Limitations and Future Work

The authors acknowledge that they did not explore contextualized embeddings like ELMO or FLAIR, which could potentially improve model performance. They suggest that implementing these advanced embeddings with deep learning models is a promising direction for future research. Additionally, the paper implies that further investigation into why traditional ML models outperform advanced ones on code-mixed data is warranted.

Summary updated

BibTeX
@inproceedings{rajalakshmi2022dlrg,
  author = {Ratnavel Rajalakshmi and Ankita Duraphe and Antonette Shibani},
  title = {{DLRG@DravidianLangTech-ACL2022: Abusive Comment Detection in Tamil using Multilingual Transformer Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure