PALI-NLP at SemEval-2022 Task 4: Discriminative Fine-tuning of Transformers for Patronizing and Condescending Language Detection

Dou Hu, Zhou Mengyuan, Xiyang Du, Mengfei Yuan, Jin Zhi, Lianxin Jiang, Mo Yang, Xiaofeng ShiNAACL 2022

TL;DR

This paper tackles the challenging task of detecting patronizing and condescending language (PCL) in text, a subtle form of harmful discourse. The authors propose a novel Transformer-based model, BERT-PCL, enhanced with two discriminative fine-tuning strategies: Grouped Layer-wise Learning Rate Decay (Grouped LLRD) and Weighted Random Sampler (WRS). Their approach achieved top rankings in the SemEval-2022 Task 4, demonstrating significant improvements over existing methods.

What Problem Does This Paper Address?

Patronizing and condescending language (PCL) is a pervasive and harmful form of communication that, while often expressed unconsciously, can perpetuate inequalities and discrimination, particularly towards vulnerable communities. Detecting PCL is a significant challenge for both human annotators and existing Natural Language Processing (NLP) systems. Unlike more overt forms of harmful language like hate speech or offensive language, PCL is characterized by its subtlety, subjectivity, and often masked kindness, making it difficult to identify. This paper addresses the need for more robust and accurate PCL detection systems, specifically within the context of the SemEval-2022 Task 4.

Methods

The core of the proposed system is a Transformer-based model, BERT-PCL, which leverages the power of pre-trained language models like BERT. To enhance its ability to capture the nuanced nature of PCL, the authors introduce two key fine-tuning strategies:

  • Grouped Layer-wise Learning Rate Decay (Grouped LLRD): Recognizing that different layers in a Transformer capture distinct types of linguistic information, this strategy applies varying learning rates to groups of layers. This allows for more controlled fine-tuning, enabling the model to better leverage both general and task-specific features. It's a variant of Layer-wise Learning Rate Decay (LLRD), but with explicit grouping of layers.
  • Weighted Random Sampler (WRS): PCL datasets are often imbalanced, with significantly fewer positive (patronizing/condescending) examples than negative ones. WRS addresses this by oversampling minority classes during training. This ensures that the model pays more attention to the crucial positive examples, leading to a more unbiased classifier that can better capture discriminative features.

For their final submission, the authors also employed a voter-based fusion technique to ensemble multiple BERT-PCL models trained with different random seeds, further boosting performance.

Datasets

The primary dataset used in this work is the Don't Patronize Me! (DPM) dataset, introduced by Pérez-Almendros et al. (2020). This dataset consists of approximately 10,469 annotated paragraphs (sentences in context) collected from English news sources, mentioning vulnerable communities.

The task was divided into two subtasks:

  • Subtask 1: Binary Classification: Identifying whether a paragraph contains any form of PCL (binary label).
  • Subtask 2: Multi-label Classification: Identifying specific categories of PCL present in a paragraph (e.g., Unbalanced power relations, Shallow solution, Authority voice, etc.).

The DPM corpus was used for training and validation, and a separate test set of around 4,000 manually annotated paragraphs was used for evaluation during the SemEval-2022 competition.

Key Insights and Ideas

  • Subtlety is Key: The paper highlights that PCL's effectiveness lies in its subtlety and often masked kindness, making it a distinct challenge from more overt forms of harmful language.
  • Layer-wise Fine-tuning Matters: The success of Grouped LLRD suggests that not all layers of a pre-trained Transformer should be treated equally during fine-tuning. Tailoring learning rates to layer groups can unlock more discriminative features.
  • Imbalance is a Real Hurdle: The significant impact of WRS underscores the critical need for strategies to handle imbalanced datasets in PCL detection. Simply ignoring the minority class leads to poor performance.
  • Ensembling for Robustness: Combining multiple well-trained models through voting further improved performance, demonstrating the benefit of diversity in model predictions.

Key Results

The proposed BERT-PCL system and its ensembles achieved impressive results at SemEval-2022 Task 4:

  • 1st place in Subtask 1 (Binary Classification).
  • 5th place in Subtask 2 (Multi-label Classification).

Specifically, on the official ranking:

  • Subtask 1: The single BERT-PCL model achieved 63.69% F1 score for the positive class, and ensembles reached up to 65.41%.
  • Subtask 2: The single BERT-PCL model achieved 43.28% macro-average F1, with ensembles reaching 45.66%.

Ablation studies confirmed the effectiveness of both Grouped LLRD and WRS, with their removal leading to performance degradation on both subtasks. The paper also showed that BERT-PCL consistently outperformed other strong baselines like BERT, ALBERT, ERNIE 2.0, and RoBERTa.

Limitations and Future Work

The authors acknowledge that detecting borderline PCL cases remains a challenge, as demonstrated by examples that even human annotators found difficult to classify. They also note that for Subtask 2, identifying multiple related PCL sub-categories simultaneously and handling the imbalance of positive PCL labels are urgent challenges.

Future work could focus on developing more sophisticated methods to address these challenges, potentially exploring more advanced techniques for capturing subtle linguistic cues and further improving the handling of imbalanced multi-label classification scenarios. The paper also suggests that effectively utilizing contextual information like country terms and search keywords is crucial for improving PCL detection.

Summary updated

BibTeX
@inproceedings{hu2022pali,
  author = {Dou Hu and Zhou Mengyuan and Xiyang Du and Mengfei Yuan and Jin Zhi and Lianxin Jiang and Mo Yang and Xiaofeng Shi},
  title = {{PALI-NLP at SemEval-2022 Task 4: Discriminative Fine-tuning of Transformers for Patronizing and Condescending Language Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure