BpHigh@TamilNLP-ACL2022: Effects of Data Augmentation on Indic-Transformer based classifier for Abusive Comments Detection in Tamil

Bhavish PahwaACL 2022

TL;DR

This paper investigates the effectiveness of data augmentation techniques for improving abusive comment detection in Tamil, a low-resource language. They found that while contextual augmentation using MURIL didn't improve performance, non-contextual augmentation with Tamil fastText significantly boosted accuracy, highlighting the importance of choosing appropriate augmentation strategies for specific language contexts.

What Problem Does This Paper Address?

The proliferation of social media has led to a surge in online harassment and abusive language. While many platforms have implemented moderation policies, these are often unsustainable due to the sheer volume of content and the psychological toll on human moderators. Automated systems are crucial, but they face significant challenges with low-resource languages like Tamil. These languages often lack sufficient labeled data, and existing multilingual models may not perform optimally. Furthermore, Tamil social media content frequently involves code-mixing (Tamil and English), adding another layer of complexity. This paper tackles the problem of building a robust abusive comment detection system for Tamil, specifically exploring how data augmentation can help overcome data scarcity and improve model performance.

Methods

The authors employ an Indic-Transformer based classifier, specifically using the MURIL Transformer as their embedding layer. MURIL is chosen because it's pre-trained on a large corpus of Indian languages, including both translated and transliterated text, making it suitable for Tamil and Tamil-English code-mixed content.

Their core contribution lies in experimenting with data augmentation techniques to address the class imbalance in their dataset (where "None of the above" is heavily overrepresented). They explore two main augmentation strategies:

  1. Contextual Word Embedding Augmentation: This method uses the MURIL Transformer itself to generate augmented sentences by replacing words based on their context.
  2. Non-Contextual Word Embedding Augmentation: This approach utilizes the Tamil fastText model from the IndicNLP suite to generate augmented sentences.

The augmented data is then used to train the MURIL-based classifier. They also detail their model architecture, which includes a classifier head built on top of the frozen MURIL layers, consisting of convolutional and dense layers with softmax activation for final predictions.

Datasets

The primary dataset used in this research is from the "Abusive Comment Detection in Tamil@DravidianLangTech-ACL 2022" shared task. This dataset consists of YouTube comments in Tamil and Tamil-English code-mixed languages. The comments are labeled with categories such as Misogyny, Misandry, Homophobia, Transphobia, Xenophobia, Counter Speech, Hope Speech, and None of the above. The dataset was split into training and development sets, and the test set was used for evaluation. The authors note that the original dataset exhibits a significant class imbalance, with the "None of the above" category being the most frequent.

Key Insights and Ideas

  • Contextual vs. Non-Contextual Augmentation Matters: The choice of augmentation technique is critical. While contextual augmentation using the same transformer model (MURIL) proved ineffective, non-contextual augmentation with a specialized Tamil embedding model (fastText) yielded significant improvements.
  • Transformer Models Aren't Always the Answer for Augmentation: The paper suggests that contextual augmentation might fail when the augmentation model and the main model are too similar (both MURIL in this case). This could be because the augmentation doesn't introduce genuinely new linguistic patterns that the pre-trained transformer hasn't already seen.
  • Potential for Incompatibility: The authors reference prior work suggesting that contextual augmentation might sometimes introduce augmentations that are incompatible with the original label's intent, potentially harming the training process. This could be a reason for the observed negative results.
  • Importance of Language-Specific Tools: The success of the Tamil fastText model highlights the value of using language-specific resources, even when working with powerful multilingual models.

Key Results

The paper presents results in Table 1, comparing different approaches on the test dataset using Macro-avg F1-score.

  • Baseline Performance: The original dataset, without any augmentation, achieved a Macro-avg F1-score of 0.17 for Tamil and 0.19 for Tamil-English.
  • Contextual Augmentation (MURIL): This approach resulted in a decrease in performance, with scores of 0.16 for Tamil and 0.13 for Tamil-English.
  • Non-Contextual Augmentation (Tamil fastText): This method showed a notable improvement, achieving 0.25 for Tamil and 0.27 for Tamil-English.

While the absolute F1-scores are modest, the relative improvement from Tamil fastText augmentation is significant, demonstrating its effectiveness in this low-resource setting.

Limitations and Future Work

Limitations:

  • Negative Impact of Contextual Augmentation: The paper explicitly notes the negative impact of contextual word-level augmentation using MURIL on performance.
  • Class Imbalance Remains a Challenge: While augmentation helped, the overall performance metrics suggest that class imbalance and the inherent difficulty of abusive language detection in low-resource, code-mixed languages remain significant challenges.

Future Work:

  • The authors suggest exploring other data augmentation techniques, potentially those that consider both label and context to avoid incompatibility issues, as proposed by Kobayashi (2018).
  • Further investigation into why contextual augmentation failed in this specific scenario, possibly by analyzing the types of linguistic patterns introduced by the augmentation.
  • Exploring more sophisticated model architectures or pre-training strategies tailored for code-mixed Indian languages.

Summary updated

BibTeX
@inproceedings{pahwa2022bphigh,
  author = {Bhavish Pahwa},
  title = {{BpHigh@TamilNLP-ACL2022: Effects of Data Augmentation on Indic-Transformer based classifier for Abusive Comments Detection in Tamil}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure