YNU-HPCC at SemEval-2022 Task 4: Finetuning Pretrained Language Models for Patronizing and Condescending Language Detection

Wenqiang Bai, Jin Wang, Xuejie ZhangNAACL 2022

TL;DR

This paper presents a system for detecting patronizing and condescending language (PCL) in text, tackling both single-label and multi-label classification tasks. By fine-tuning the powerful BERT model with a custom classification layer, the system achieved competitive results in the SemEval-2022 competition, demonstrating BERT's effectiveness for nuanced language understanding.

What Problem Does This Paper Address?

The paper addresses the challenge of automatically identifying patronizing and condescending language (PCL) in text. This is a nuanced form of communication that can be subtle and context-dependent, making it difficult for traditional NLP methods to detect. The SemEval-2022 Task 4 specifically aimed to develop systems capable of recognizing PCL, with two subtasks: first, identifying if a sentence contains any PCL, and second, identifying which specific types of PCL are present. This task is crucial for applications like content moderation, understanding user sentiment in online interactions, and building more empathetic AI systems.

Methods

The core of the YNU-HPCC system is the Bidirectional Encoder Representations from Transformers (BERT) model. BERT, a pre-trained language model known for its strong performance on various NLP tasks, was chosen as the base architecture.

For both subtasks, the authors employed a fine-tuning approach. This involves taking the pre-trained BERT model and further training it on the specific PCL detection dataset.

The key modification to the standard BERT architecture was the addition of a fully connected layer at the end.

  • For Subtask 1 (single-label classification), this layer, combined with a softmax activation and argmax function, was used to output a binary classification (PCL present or not).
  • For Subtask 2 (multi-label classification), the fully connected layer was designed to output a vector representing the probability of each of the seven PCL categories being present. A sigmoid activation function was then applied to each element of this vector, followed by a rounding rule to determine the final labels.

The authors also detail standard text preprocessing steps, including removing HTML tags and adding a special [CLS] token to the beginning of each input sequence, which is a common practice when using BERT for classification.

Datasets

The system was trained and evaluated using datasets provided for the SemEval-2022 Task 4 competition:

  • dontpatronizeme_pcl.tsv: This dataset was used for the single-label classification task. It contains paragraphs annotated with labels ranging from 0 to 4. For the competition, these labels were re-mapped to binary (0 for no PCL, 1 for PCL).
  • dontpatronizeme_categories.tsv: This dataset was used for the multi-label classification task. It contains paragraphs annotated with 7 labels, indicating the presence or absence of specific PCL categories (e.g., Unbalanced_power_relations, Shallow_solution).

No new datasets were introduced by the authors; they utilized the official competition benchmarks.

Key Insights and Ideas

  • BERT's Adaptability: The paper highlights BERT's remarkable ability to be fine-tuned for highly specific and nuanced NLP tasks like PCL detection, even with relatively small labeled datasets.
  • Unified Approach for Subtasks: The authors demonstrate that a single base model (BERT) can be effectively adapted for both single-label and multi-label classification by simply modifying the output layer and activation functions.
  • Importance of Pre-trained Models: The success of the system is attributed to the power of pre-trained language models, which leverage massive amounts of unlabeled data to learn general language understanding before being specialized for a downstream task. This significantly reduces the need for extensive manual feature engineering.

Key Results

The system achieved "relatively good results" on the competition's official leaderboard.

  • Subtask 1 (Single-label Classification): The system ranked 42/81 in precision, 47/81 in recall, and 52/81 in F1 score.
  • Subtask 2 (Multi-label Classification): The system achieved rankings between 24th and 35th out of 81 participants for the seven individual PCL labels.

While specific accuracy numbers are not explicitly stated for the overall task, the rankings indicate a competitive performance against other state-of-the-art approaches.

Limitations and Future Work

  • Limitations: The paper does not explicitly detail limitations of their current system, beyond acknowledging that the results are "relatively good" rather than top-ranked. The reliance on a single pre-trained model (BERT) might also be a limitation if other architectures could offer superior performance for this specific task.
  • Future Work: The authors express a desire to "explore other models and methods in the sentiment analysis field" for future work, suggesting an interest in investigating alternative architectures or ensemble methods to further improve PCL detection capabilities.

Summary updated

BibTeX
@inproceedings{bai2022ynu,
  author = {Wenqiang Bai and Jin Wang and Xuejie Zhang},
  title = {{YNU-HPCC at SemEval-2022 Task 4: Finetuning Pretrained Language Models for Patronizing and Condescending Language Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure