Optimize_Prime@DravidianLangTech-ACL2022: Abusive Comment Detection in Tamil

Shantanu Patankar, Omkar Gokhale, Onkar Litake, Aditya Mandke, Dipali KadamACL 2022

TL;DR

This paper tackles the challenge of detecting abusive comments in Tamil, a low-resource language, by exploring various machine learning models, including Transformers and RNNs. They achieved promising results, with Transformer models like MuRIL and XLM-ROBERTa performing best on both Tamil and code-mixed Tamil-English datasets, highlighting the effectiveness of pre-trained models for this task.

What Problem Does This Paper Address?

The proliferation of social media has unfortunately led to an increase in online abuse, including hate speech and cyberbullying. Detecting and classifying these abusive comments is crucial for mitigating their harmful effects. However, many languages, particularly those in the Dravidian language family like Tamil, are considered "low-resource," meaning they lack sufficient annotated data for training robust natural language processing (NLP) models. This paper specifically addresses the problem of abusive comment detection in Tamil and Tamil-English code-mixed text, aiming to build systems that can accurately identify and categorize such offensive content.

Methods

The authors employed a multi-pronged approach, experimenting with three main categories of models:

  • Ensemble Models: They combined a powerful pre-trained Transformer model, Multilingual BERT (M-BERT), with various traditional machine learning classifiers (e.g., Logistic Regression, Support Vector Machines, Multi-Layer Perceptron). This approach leverages M-BERT's contextual understanding and then uses simpler models for classification.
  • Recurrent Neural Networks (RNNs): They utilized Long Short-Term Memory (LSTM) networks as a baseline and also explored ULM-FiT, a transfer learning approach that fine-tunes language models for specific tasks.
  • Transformer Models: This was a core focus, with the team experimenting with several state-of-the-art Transformer architectures:
  • MuRIL: A language model specifically designed for Indian languages.
  • XLM-ROBERTa: A multilingual Transformer pre-trained on a vast amount of text across many languages.
  • M-BERT: A widely used multilingual BERT model.
  • Indic BERT: A multilingual ALBERT model trained on Indian languages.

Data pre-processing involved cleaning text (removing punctuation, URLs, stop words) and replacing emojis with their textual equivalents. To address class imbalance, they experimented with over-sampling, under-sampling, SMOTE, and class weights.

Datasets

The paper utilizes the dataset from the ACL 2022 shared task "Abusive Comment Detection in Tamil." This dataset consists of YouTube comments in two formats:

  • Tamil Data: Comments written purely in Tamil.
  • Tamil-English Codemixed Data: Comments that interleave Tamil and English words.

The dataset includes labels for nine categories: Misandry, Counter-speech, Misogyny, Xenophobia, Hope-Speech, Homophobia, Transphobic, Not-Tamil, and None-of-the-above. Notably, the 'Not-Tamil' label had no test or development data, so classification was performed on the remaining eight labels.

Key Insights and Ideas

  • Transformer Dominance: Pre-trained Transformer models, especially those tailored for multilingual or Indian language contexts (MuRIL, XLM-ROBERTa), significantly outperform traditional RNNs and simpler ensemble methods for this task.
  • Data Cleaning Nuance: While data cleaning generally improved RNN performance, it surprisingly degraded the performance of MuRIL. This suggests that for highly sophisticated pre-trained models, raw or minimally processed data might retain more valuable contextual information.
  • Code-Mixing Complexity: The code-mixed Tamil-English data presented a unique challenge, and models that handled multilingualism well (like MuRIL and M-BERT) showed strong performance.
  • Class Imbalance Strategies: Over-under sampling proved effective for Transformer and ensemble models, while class weights boosted the performance of an M-BERT-Logistic Regression ensemble. This highlights that different imbalance handling techniques are suited to different model types.

Key Results

The paper reports the following key findings:

  • Tamil Data: MuRIL and XLM-ROBERTa achieved the best results with a macro-averaged F1 score of 0.43.
  • Tamil-English Codemixed Data: MuRIL and M-BERT delivered the top performance, achieving a macro-averaged F1 score of 0.45.
  • Among ensemble models, Support Vector Machine (SVM) performed best on Tamil data (macro F1: 0.33), while Multi-Layer Perceptron (MLP) was best on codemixed data (macro F1: 0.35).
  • ULM-FiT, when fine-tuned on codemixed data, showed the highest macro F1 score among RNN models (0.40).

Limitations and Future Work

Limitations:

  • The 'Not-Tamil' class was excluded from classification due to a lack of data, meaning the system doesn't explicitly identify non-Tamil comments.
  • The performance of data cleaning varied across model types, indicating that optimal pre-processing might be model-dependent.

Future Work:

The authors suggest exploring genetic algorithm-based ensembling methods to further boost model performance. Additionally, investigating more advanced techniques for handling low-resource languages and code-mixed text could lead to even more robust abusive comment detection systems.

Summary updated

BibTeX
@inproceedings{patankar2022optimize,
  author = {Shantanu Patankar and Omkar Gokhale and Onkar Litake and Aditya Mandke and Dipali Kadam},
  title = {{Optimize\_Prime@DravidianLangTech-ACL2022: Abusive Comment Detection in Tamil}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure