SSNCSE NLP@TamilNLP-ACL2022: Transformer based approach for detection of abusive comment for Tamil language

Bharathi B, Josephine VarshaACL 2022

TL;DR

This paper tackles the challenge of detecting abusive comments in Tamil, a low-resource language, using transformer-based models. The authors adapted pre-trained models like BERT and XLNet to classify Tamil and code-mixed Tamil-English comments, achieving promising results and ranking well in a shared task.

What Problem Does This Paper Address?

The proliferation of social media has unfortunately led to a surge in abusive behavior online, including misogyny, misandry, homophobia, and cyberbullying. Identifying and moderating these harmful comments is a monumental task for human moderators, especially in languages with limited resources. Tamil, a classical language with a rich literary history, faces a particular challenge due to the scarcity of readily available datasets and NLP tools for tasks like abusive comment detection. This paper aims to bridge this gap by developing and evaluating transformer-based models for identifying abusive content in Tamil and code-mixed Tamil-English text.

Methods

The core of this research lies in leveraging pre-trained transformer models, which have revolutionized NLP tasks. The authors experimented with three prominent models:

  • BERT (Bidirectional Encoder Representations from Transformers): A powerful model known for its ability to understand context from both left and right directions in a sentence.
  • XLNet: An autoregressive model that builds upon BERT's strengths by incorporating a permutation-based training objective, aiming to overcome some of BERT's limitations.
  • m-BERT (Multilingual BERT): A version of BERT pre-trained on a large corpus of over 100 languages, making it suitable for multilingual tasks.

The process involved:

  1. Data Pre-processing: This crucial step involved cleaning the raw text data. Common pre-processing steps included removing hashtags, HTML tags, mentions, and URLs, annotating emojis, converting text to lowercase, expanding abbreviations, removing special and accented characters, reducing lengthened words, and removing extra white spaces. The re module in Python was used for text cleaning, and RegexpTokenizer for tokenization.
  2. Model Training and Evaluation: The pre-trained models were fine-tuned on the provided datasets for the specific task of abusive comment detection. The performance of these models was then evaluated using metrics like precision, recall, and F1-score.

Datasets

The study utilized datasets provided by the LT-EDI 2021 organizers for the shared task on Abusive Comment Detection. These datasets comprised:

  • Tamil Text: Containing comments exclusively in Tamil.
  • Tamil-English Code-Mixed Text: Containing comments that blend Tamil and English.

The datasets were split into training, development, and test sets. The specific sizes are detailed in Table 1:

Language Training Development Test
Tamil 2240 560 700
Tamil-English 5948 1488 1859

Key Insights and Ideas

  • Transformer Models Excel in Low-Resource Languages: Despite the inherent challenges of low-resource languages, transformer models, particularly BERT, demonstrated strong performance in detecting abusive comments in Tamil and code-mixed Tamil-English.
  • Code-Mixing is a Significant Challenge: The performance on the Tamil-English code-mixed dataset was notably higher than on the pure Tamil dataset, suggesting that the presence of English words aids the models in understanding context, or that the Tamil-only dataset contains more nuanced or context-dependent abusive language.
  • Pre-trained Models are a Powerful Starting Point: The success of fine-tuning pre-trained models highlights their effectiveness as a foundation for NLP tasks, even in languages with limited resources. This approach significantly reduces the need for extensive feature engineering.
  • Subjectivity and Context Matter: The paper acknowledges that abusive language is often subjective and context-dependent, making perfect detection a difficult goal. This is reflected in the error analysis, where confusion matrices reveal misclassifications between different categories.

Key Results

The study reported the following key results:

  • Tamil-English Code-Mixed Dataset: The bert-base-uncased model achieved the best weighted average F1-score of 0.96.
  • Tamil Dataset: The bert-base-uncased model achieved a weighted average F1-score of 0.59.
  • Rankings: The team secured the 8th rank in the shared task for the code-mixed Tamil dataset and the 12th rank for the Tamil dataset.

The paper also includes confusion matrices (Figure 2 and 3) that provide a detailed breakdown of classification performance across different abusive categories for both datasets.

Limitations and Future Work

The authors acknowledge that their adopted models did not achieve a perfect F1 score of 1, indicating room for improvement. The paper suggests that performance can be further enhanced by adopting more favorable features. While not explicitly detailed, future work could explore:

  • More Sophisticated Pre-processing: Investigating advanced techniques for handling code-mixed text.
  • Ensemble Methods: Combining the predictions of multiple models to potentially improve robustness.
  • Data Augmentation: Generating more training data to address the low-resource nature of Tamil.
  • Contextual Information: Incorporating more explicit contextual information beyond the immediate comment, such as conversation threads.

Summary updated

BibTeX
@inproceedings{b2022ssncse,
  author = {Bharathi B and Josephine Varsha},
  title = {{SSNCSE NLP@TamilNLP-ACL2022: Transformer based approach for detection of abusive comment for Tamil language}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure