CS/NLP at SemEval-2022 Task 4: Effective Data Augmentation Methods for Patronizing Language Detection and Multi-label Classification with RoBERTa and GPT3

Daniel Saeedi, Sirwe Saeedi, Aliakbar Panahi, Alvis C.m. FongNAACL 2022

TL;DR

This paper tackles the challenge of detecting patronizing and condescending language (PCL) in text, a nuanced task that goes beyond simple sentiment analysis. The authors propose effective data augmentation techniques to improve the performance of transformer-based models like RoBERTa and GPT-3, achieving competitive results in the SemEval-2022 PCL detection competition.

What Problem Does This Paper Address?

Natural Language Understanding (NLU) has seen remarkable progress with deep learning, but understanding subtle human intentions and attitudes, especially in the context of marginalized communities, remains a significant challenge. Patronizing and condescending language (PCL) is one such area. PCL involves projecting a sense of superiority, which can lead to feelings of unfair treatment and inadequacy in vulnerable populations. Unlike sentiment analysis, PCL detection requires understanding context and world knowledge, making it inherently difficult. Furthermore, PCL datasets are often imbalanced, with a vast majority of non-PCL examples, making it hard for models to learn to identify the minority PCL class. This paper addresses these challenges by focusing on effective data augmentation strategies to improve PCL detection and multi-label classification.

Methods

The core of this paper lies in its exploration of various data augmentation (DA) techniques to combat data imbalance and enhance model performance. The authors employed:

  • Synonym Replacement (SR): A straightforward method of replacing non-stop words with their synonyms. While lightweight, it didn't significantly boost performance on its own.
  • Oversampling (OS): Randomly duplicating minority class samples to balance the dataset. This showed a notable improvement.
  • Back Translation (BT): Translating text to another language and then back to the original. This technique, while promising, led to overfitting in their experiments.
  • PEGASUS Paraphrasing: A modified version of the PEGASUS model was developed to handle arbitrarily long documents, overcoming the Transformer's typical sequence length limitations. This "multi-sentence PEGASUS" was used to generate paraphrased versions of the PCL dataset, significantly increasing its size.

These DA methods were used to augment the training data for fine-tuning pre-trained RoBERTa and GPT-3 models.

Datasets

The primary dataset used in this work is from the SemEval-2022 Task 4: Patronizing and Condescending Language Detection competition. This dataset includes:

  • Binary Classification Data: Human-labeled paragraphs for detecting the presence of PCL.
  • Multi-label Classification Data: Paragraphs where PCL instances can belong to multiple categories (e.g., "unbalanced power relations," "shallow solution," "authority voice").

The authors highlight the significant class imbalance in this dataset, with non-PCL examples making up approximately 90.4% of the binary classification data.

Key Insights and Ideas

  • PCL is More Than Sentiment: The paper strongly emphasizes that PCL detection is distinct from sentiment analysis. Positive or negative sentiment in a sentence doesn't directly correlate with whether it's patronizing.
  • Data Imbalance is a Major Hurdle: The extreme imbalance in PCL datasets significantly hinders model performance, leading to a bias towards predicting the majority (non-PCL) class.
  • Data Augmentation is Crucial: Combining multiple DA techniques, particularly PEGASUS paraphrasing with other methods, proved effective in boosting performance and achieving competitive results.
  • Large Language Models (LLMs) Show Promise with Few-Shot Learning: GPT-3 models (Davinci and Curie) demonstrated surprisingly good performance on PCL detection with minimal labeled examples (few-shot learning), suggesting their potential for low-resource NLP tasks.
  • PEGASUS Modification for Long Texts: The development of "multi-sentence PEGASUS" is a key technical contribution, enabling the model to handle and paraphrase long documents, which is essential for many real-world text analysis tasks.

Key Results

  • The proposed system ranked 16th out of 79 NLP research teams in the SemEval-2022 PCL detection competition.
  • The system achieved an 80% F1-score on the training dataset and 58% on the test dataset. This is noteworthy, as the winning system achieved a 65% F1-score.
  • Fine-tuning RoBERTa with a combination of SR, OS, and BT DA methods boosted performance to 81% F1-score.
  • GPT-3 models (Davinci and Curie) performed well on the PCL binary classification task using few-shot learning, achieving 65% and 63% F1-scores respectively, without extensive hyper-parameter tuning.

Limitations and Future Work

  • Overfitting with Back Translation: The paper notes that Back Translation, while a powerful technique, led to significant overfitting in their experiments.
  • GPT-3 Cost: While GPT-3 models show promise, their cost for training and inference is a significant barrier, limiting their practical application for many researchers.
  • Future Research Directions: The authors suggest exploring GPT-3's applications for PCL detection and multi-label classification tasks further, despite the cost. They also believe there is significant potential for further improvements in this area.

Summary updated

BibTeX
@inproceedings{saeedi2022cs,
  author = {Daniel Saeedi and Sirwe Saeedi and Aliakbar Panahi and Alvis C.M. Fong},
  title = {{CS/NLP at SemEval-2022 Task 4: Effective Data Augmentation Methods for Patronizing Language Detection and Multi-label Classification with RoBERTa and GPT3}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure