Sapphire at SemEval-2022 Task 4: A Patronizing and Condescending Language Detection Model Based on Capsule Networks

Sihui Li, Xiaobing ZhouNAACL 2022

TL;DR

This paper presents Team Sapphire's approach to detecting patronizing and condescending language (PCL) in news texts. They leverage BERT for text encoding and capsule networks for feature extraction, achieving a competitive F1 score of 0.5187 on the SemEval-2022 Task 4 dataset. This work highlights the effectiveness of combining powerful pre-trained language models with specialized neural architectures for nuanced NLP tasks.

What Problem Does This Paper Address?

The paper tackles the challenge of identifying "Patronizing and Condescending Language" (PCL) in media texts. PCL is subtle language that can subtly demean others, often with a seemingly well-intentioned facade. While users might overlook these elements, they can contribute to societal stereotypes, normalize discrimination, and lead to exclusion. Detecting PCL is crucial for promoting more inclusive and responsible communication by alerting speakers to examine their language. This task, specifically Subtask 1 of SemEval-2022 Task 4, frames PCL detection as a binary classification problem: determining if a news text contains PCL or not.

Methods

Team Sapphire's approach, dubbed "BERT-Caps," combines two key components:

  • BERT-base for Embedding and Encoding: They utilize the BERT-base model as a pre-trained language model. BERT excels at generating contextual word embeddings, capturing the meaning of words within their surrounding text. The input news text is preprocessed (lowercase, adding [CLS] and [SEP] tokens) and then fed into BERT to obtain vector representations.
  • Capsule Networks for Feature Extraction: Following BERT's encoding, capsule networks are employed to extract features from the BERT output. Capsule networks, an advancement over traditional Convolutional Neural Networks (CNNs), are designed to better preserve spatial relationships and hierarchical structures. In this context, they are used to capture more nuanced features from the encoded text vectors. A dynamic routing algorithm is used within the capsule layer.
  • Dropout and Fully Connected Layer for Prediction: To prevent overfitting and improve generalization, dropout is applied to the combined output of the capsule layer and the BERT embeddings. Finally, a fully connected layer with a softmax activation function predicts the probability of the text containing PCL. Categorical cross-entropy is used as the loss function.

Datasets

The primary dataset used for training and evaluation is the SemEval-2022 Task 4: Patronizing and Condescending Language Detection dataset. For Subtask 1, the dataset is framed as a binary classification problem where PCL levels 0-1 are considered negative examples, and levels 2-4 are positive examples. The dataset is imbalanced, with approximately a 10:1 ratio of negative to positive labels. The official evaluation metric for this subtask is the F1 score over the positive class, which is particularly important for imbalanced datasets.

Key Insights and Ideas

  • Synergy of BERT and Capsule Networks: The paper demonstrates that combining the powerful contextual understanding of BERT with the feature extraction capabilities of capsule networks can yield effective results for a nuanced task like PCL detection.
  • Importance of Capsule Networks: The results suggest that capsule networks contribute significantly to performance. Removing them led to a notable decrease in the model's effectiveness.
  • Impact of Dropout: While dropout is a standard regularization technique, the paper notes that its removal led to a less significant performance drop compared to removing capsule networks, implying capsule networks might be more critical for this specific task.
  • BERT-Caps Outperforms Other BERT Variants: The BERT-Caps model, when compared to other BERT-based models (like RoBERTa-Caps and DeBERTa-Caps) with similar capsule network architectures, performed better in their experimental setup.

Key Results

The BERT-Caps model achieved an F1 score of 0.5187 on the test set for Subtask 1. While this score is competitive, it's noted that the top-performing teams achieved F1 scores above 0.6400, indicating room for improvement. Team Sapphire's final ranking was 35th. Table 2 provides a detailed comparison of different model configurations:

  • BERT-Caps: Precision 0.5935, Recall 0.4606, F1 score 0.5187
  • BERT-Caps w/o dropout: Precision 0.6, Recall 0.4542, F1 score 0.5170
  • BERT-Caps w/o capsule networks: Precision 0.5782, Recall 0.4195, F1 score 0.4862
  • ROBERTa_baseline: Precision 0.3935, Recall 0.6530, F1 score 0.4911

Limitations and Future Work

The authors acknowledge that their model's performance, while good, is still a gap from the top-ranked teams. They suggest several avenues for future work:

  • Exploring other text representations: Investigating alternative ways to represent text could potentially improve performance.
  • Adjusting the loss function weights: Given the dataset's imbalance, fine-tuning the loss function's weighting for different classes might yield better results.
  • Further experimentation with BERT variants: While BERT-Caps performed best among their tested BERT-based models, continued exploration of other pre-trained models could be beneficial.
  • Hyperparameter tuning: The paper mentions trying variations in capsule numbers and hidden neurons without significant improvement, but further systematic hyperparameter optimization could still be valuable.

Summary updated

BibTeX
@inproceedings{li2022sapphire,
  author = {Sihui Li and Xiaobing Zhou},
  title = {{Sapphire at SemEval-2022 Task 4: A Patronizing and Condescending Language Detection Model Based on Capsule Networks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure