Hypers@DravidianLangTech-EACL2021: Offensive language identification in Dravidian code-mixed YouTube Comments and Posts

Charangan Vasantharajan, Uthayasanker Thayasivam2021

TL;DR

This paper tackles the challenging problem of identifying offensive language in code-mixed Dravidian languages (Tamil, Kannada, Malayalam) found on YouTube. By fine-tuning a pre-trained multilingual BERT model, the authors achieved competitive results, ranking third overall in a shared task and demonstrating BERT's effectiveness for low-resource, code-mixed text.

What Problem Does This Paper Address?

Social media has become a primary channel for communication, with users frequently mixing languages within their posts and comments – a phenomenon known as code-mixing. This is particularly prevalent in low-resource languages like Tamil, Kannada, and Malayalam, where users might blend their native tongue with English. Identifying offensive content within this code-mixed text is crucial for various applications, from content moderation to understanding online discourse. However, the scarcity of labeled datasets and the inherent complexity of code-mixed language pose significant challenges for traditional natural language processing (NLP) models, which are often trained on monolingual data. This paper aims to bridge this gap by developing an effective offensive language identification system for these Dravidian code-mixed languages.

Methods

The core of this research lies in leveraging the power of pre-trained transformer models. The authors adopted a bidirectional approach using BERT (Bidirectional Encoder Representations from Transformers), specifically the bert-base-multilingual-cased model. This model is pre-trained on a massive corpus of text from over 100 languages, allowing it to capture contextual relationships within sentences effectively.

The key technique employed is fine-tuning. This involves taking the pre-trained BERT model and further training it on the specific task of offensive language identification using the collected Dravidian code-mixed datasets. This transfer learning approach allows the model to adapt its general language understanding capabilities to the nuances of offensive language in the target languages.

Before feeding the data into BERT, two preprocessing steps were applied:

  1. Emoji Conversion: Emojis, which are common in social media, were converted into text representations to preserve their emotional context.
  2. Punctuation Removal: Punctuation marks were removed from the input text to simplify processing.

The BERT model's architecture involves an encoder that reads the input text and a decoder that makes predictions. For this task, the model was configured for text classification, outputting one of six predefined labels.

Datasets

The study utilized existing, manually annotated datasets for Tamil-English, Kannada-English, and Malayalam-English code-mixed content. These datasets were provided by organizers of the EACL 2021 shared task on Offensive Language Identification in Dravidian Languages. The paper notes that the Tamil dataset was significantly larger than the Kannada and Malayalam datasets. The data was split into training and validation sets with an 8:2 ratio for training.

Key Insights and Ideas

  • BERT's Efficacy in Low-Resource Code-Mixing: The study strongly suggests that pre-trained multilingual BERT models, when fine-tuned, are highly effective for offensive language identification even in low-resource, code-mixed scenarios. This overcomes the limitations of traditional unidirectional models.
  • The Challenge of Data Imbalance: Despite the larger size of the Tamil dataset, its F1-score was lower than Malayalam's. This highlights the potential impact of data imbalance and the need for strategies to address it, even when using powerful models.
  • Importance of Contextual Understanding: The bidirectional nature of BERT allows it to understand context from both left-to-right and right-to-left, which is crucial for deciphering the often subtle nuances of offensive language in code-mixed text.
  • Multilingual Models Outperform Monolingual for Code-Mixing: The success of the multilingual BERT model in this code-mixed task underscores the advantage of using models trained on diverse linguistic data when dealing with language mixing.

Key Results

The proposed BERT-based system achieved the following weighted average F1-scores on the test set:

  • Malayalam: 0.96 (Ranked 2nd out of 30 participants)
  • Tamil: 0.73 (Ranked 6th out of 31 participants)
  • Kannada: 0.70 (Ranked 5th out of 29 participants)

Overall, the "Hypers" team ranked 3rd among all submitted systems in the shared task, demonstrating the model's strong performance across all three languages. The authors note that their F1-scores were very close to the top-ranked teams, indicating the competitiveness of their approach.

Limitations and Future Work

The authors acknowledge several limitations and propose future directions:

  • Data Imbalance: The significant imbalance in dataset sizes across languages, particularly the smaller datasets for Kannada and Malayalam, might have impacted performance.
  • Potential for Further Improvement: Despite good results, the authors believe performance can be further enhanced.
  • Future Work:
    • Character-Level Embeddings: Incorporating character-level embeddings alongside word embeddings to better represent out-of-vocabulary (OOV) words and misspelled terms.
  • Handling Imbalanced Data: Employing oversampling techniques to increase the representation of minority classes in the training data.
  • Exploring Other Approaches: Investigating alternative approaches to further boost model performance.

Summary updated

Abstract

Code-Mixed Offensive contents are used pervasively in social media posts in the last few years. Consequently, gained the significant attraction of the research community for identifying the different forms of such content (e.g., hate speech, and sentiments) and contributed to the creation of datasets. Most of the recent studies deal with high-resource languages (e.g., English) due to many publicly available datasets, and by the lack of dataset in low-resource anguages, those studies are slightly involved in these languages. Therefore, this study has the focus on offensive language identification on code-mixed low-resourced Dravidian languages such as Tamil, Kannada, and Malayalam using the bidirectional approach and fine-tuning strategies. According to the leaderboard, the proposed model got a 0.96 F1-score for Malayalam, 0.73 F1-score for Tamil, and 0.70 F1-score for Kannada in the bench-mark. Moreover, in the view of multilingual models, this modal ranked 3rd and achieved favorable results and confirmed the model as the best among all systems submitted to these shared tasks in these three languages.
BibTeX
@misc{vasantharajan2021hypers,
  author = {Charangan Vasantharajan and Uthayasanker Thayasivam},
  title = {{Hypers@DravidianLangTech-EACL2021: Offensive language identification in Dravidian code-mixed YouTube Comments and Posts}},
  year = {2021},
}
Figure