GUTS at SemEval-2022 Task 4: Adversarial Training and Balancing Methods for Patronizing and Condescending Language Detection

Junyu Lu, Hao Zhang, Tongyue Zhang, Hongbo Wang, Haohao Zhu, Bo Xu, Hongfei LinNAACL 2022

TL;DR

This paper tackles the challenging task of detecting patronizing and condescending language (PCL) in text, which often carries subtle toxicity and can reinforce harmful stereotypes. The authors introduce an approach combining adversarial training for robustness and novel loss functions to handle class imbalance, achieving competitive results in the SemEval-2022 Task 4.

What Problem Does This Paper Address?

Patronizing and condescending language (PCL) is a subtle form of toxic speech that, despite often being used with good intentions, can have harmful effects by perpetuating stereotypes and increasing exclusion. Detecting PCL is difficult because its toxicity is often masked by a seemingly charitable or compassionate tone. This paper addresses the challenge of automatically identifying PCL in media text, a task that requires nuanced understanding beyond simple sentiment analysis. The SemEval-2022 Task 4 specifically aimed to build systems capable of detecting PCL and its various subtypes.

Methods

The authors employ a two-pronged approach for the two subtasks of SemEval-2022 Task 4:

  • Subtask A (Binary Classification): For detecting PCL in general, they utilize a pre-trained language model (like RoBERTa) and enhance its robustness through adversarial training. Specifically, they use the Fast Gradient Method (FGM) to introduce small perturbations to the sentence embeddings, forcing the model to learn more resilient representations. The overall loss function combines the standard loss on original data with the loss on adversarially perturbed data.
  • Subtask B (Multi-label Classification): This subtask involves identifying one of seven specific PCL subtypes. A major challenge here is the long-tailed distribution of labels, meaning some subtypes are much rarer than others. To combat this class imbalance, they experiment with several improved multi-label cross-entropy loss functions:
  • Focal Loss (FL): Down-weights easy examples and focuses training on hard-to-classify ones.
  • Class-balanced Focal Loss (CB): Re-balances loss based on the effective number of samples per class.
  • Distribution-balanced Loss (DB): Addresses label co-occurrence imbalance. They also explore incorporating Negative Tolerant Regularization (NTR) with these loss functions to mitigate over-suppression of negative labels.

Datasets

The primary dataset used is from SemEval-2022 Task 4, which consists of annotated paragraphs extracted from news articles. This dataset was originally introduced by Pérez-Almendros et al. (2022) and contains examples labeled for the presence of PCL (Subtask A) and for one of seven specific PCL subtypes (Subtask B). The training set comprises 10,469 samples, and the final test set has 2,971 samples.

Key Insights and Ideas

  • Adversarial training significantly improves robustness: The paper demonstrates that even small, carefully crafted perturbations can reveal model weaknesses. Adversarial training helps the model generalize better and become more resilient to variations in input, which is crucial for subtle language detection.
  • Class imbalance is a major hurdle in PCL subtype detection: The long-tailed distribution of PCL subtypes necessitates specialized loss functions beyond standard cross-entropy to prevent models from simply predicting the majority classes.
  • Balancing methods offer substantial gains: The experiments show that using Focal Loss, Class-balanced Focal Loss, and Distribution-balanced Loss significantly outperforms a baseline approach, highlighting the importance of addressing data imbalance.
  • PCL is linked to positive emotions and higher intensity: An interesting secondary analysis suggests that PCL is more likely to appear in text expressing positive emotions (perhaps due to "good intentions") and in text with higher emotional intensity, where speakers might adopt a stronger tone.

Key Results

  • Subtask A: The system achieved official rankings of 17/79 in the evaluation. Adversarial training with FGM improved the F1 score from 0.5699 to 0.5785 during evaluation and from 0.5545 to 0.5790 during testing.
  • Subtask B: The system ranked 16/49. The best-performing models for Subtask B were trained with FL+NTR and CB+NTR, achieving macro-average F1 scores of 0.3917 and 0.3922 respectively during evaluation. The FL+NTR model performed better on the final test set.

Limitations and Future Work

  • Limitations: The authors note that the effectiveness of Class-balanced Focal Loss can be sensitive to the assumed sample space size (β). If there's a significant mismatch between training and test set label distributions, its performance might degrade.
  • Future Work: The authors propose exploring multi-task learning for PCL detection, specifically by jointly predicting sentiment polarity. They also plan to further investigate the relationship between sentiment analysis and PCL detection, potentially developing multitasking frameworks to leverage these connections.

Summary updated

BibTeX
@inproceedings{lu2022guts,
  author = {Junyu Lu and Hao Zhang and Tongyue Zhang and Hongbo Wang and Haohao Zhu and Bo Xu and Hongfei Lin},
  title = {{GUTS at SemEval-2022 Task 4: Adversarial Training and Balancing Methods for Patronizing and Condescending Language Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure