University of Bucharest Team at Semeval-2022 Task4: Detection and Classification of Patronizing and Condescending Language

Tudor Dumitrascu, Raluca-Andreea Gînga, Bogdan Dobre, Bogdan Radu Silviu SieleckiNAACL 2022

TL;DR

This paper presents the University of Bucharest team's approach to detecting and classifying patronizing and condescending language (PCL) in text for SemEval-2022 Task 4. They explored various machine learning and deep learning models, ultimately finding that BERT-based models, particularly BertForSequenceClassification, achieved the best performance for binary classification, while multi-label classification proved more challenging.

What Problem Does This Paper Address?

The paper tackles the challenge of identifying and categorizing "Patronizing and Condescending Language" (PCL) in text, specifically when it refers to vulnerable communities. This type of language, often subtle and embedded in news articles, can perpetuate unfair treatment and stereotypes. The task, as defined by SemEval-2022 Task 4, is divided into two subtasks:

  1. Binary Classification: Determining if a given paragraph contains any PCL.
  2. Multi-label Multi-class Classification: Identifying which specific PCL categories (e.g., "Unbalanced power relations," "Shallow solution," "Presupposition") are used to express condescension.

The authors highlight that PCL detection is crucial for promoting fairer media representation and understanding nuanced forms of linguistic bias.

Methods

The team experimented with a wide range of methods, from traditional machine learning to state-of-the-art deep learning models.

  • Preprocessing: Standard text cleaning techniques were applied, including lowercasing, tokenization, stopword removal, and filtering short words. They also explored lemmatization and stemming.
  • Feature Extraction: Various feature extraction methods were employed, such as TF-IDF, Word2Vec (including Skip-Gram), and BERT tokenizers.
  • Machine Learning Models:
    • Classical ML: Logistic Regression, Random Forest, Support Vector Machines, and XGBoost were tested.
  • Neural Networks: Simple neural networks with dense layers and Long Short-Term Memory (LSTM) networks were implemented.
  • Deep Learning Models:
    • BERT-based Models: Several pre-trained BERT models (e.g., BERT Uncased, BERT Multilingual Uncased, BERT HateXplain, DistilBERT, DistilRoBERTa) were fine-tuned for the task. BertForSequenceClassification was a key model used.
  • Universal Sentence Encoder (USE): This model was also explored, particularly in the "Text shards" approach.
  • Handling Imbalance: Given the imbalanced nature of the dataset (especially for binary classification), the authors employed techniques like:
  • Class Weighting: Assigning different weights to classes to penalize misclassifications of the minority class.
  • Oversampling/Undersampling: Methods like SMOTE, Adasyn, SVM-SMOTE, and SPE were used to balance the dataset.
  • Data Augmentation: They attempted to augment the dataset by collecting hate speech datasets from Kaggle.
  • "Text Shards" Approach: For multi-class classification, they proposed a novel approach where only the actual PCL segments within a text were used for training, aiming to mitigate confusion caused by mixed PCL and non-PCL content. This involved sliding a window over the text to identify PCL substrings.

Datasets

The primary dataset used for this task was "Don't Patronize Me!" (Pérez-Almendros et al., 2020). This dataset contains news article paragraphs mentioning vulnerable communities, manually annotated for the presence of PCL and its specific categories.

  • Subtask 1 (Binary Classification): Used approximately 10,636 paragraphs.
  • Subtask 2 (Multi-label Multi-class Classification): Used approximately 2,792 instances, split according to PCL categories.

The authors also mention using external hate speech datasets from Kaggle for data augmentation, though this approach did not yield good results.

Key Insights and Ideas

  • BERT Dominance: BERT-based models, particularly BertForSequenceClassification, emerged as the clear winners for binary PCL detection, significantly outperforming traditional ML and simpler neural networks. This highlights the power of pre-trained language models for understanding nuanced linguistic phenomena.
  • Imbalance is a Major Hurdle: The dataset's severe class imbalance for binary classification was a significant challenge. While various techniques were employed, achieving high performance on the minority class remained difficult.
  • Multi-label Classification is Tough: Identifying specific PCL categories proved much harder than binary detection. The models struggled to learn all the classes, and even general class approaches didn't yield substantial improvements. This suggests that PCL is complex and context-dependent.
  • Data Augmentation Limitations: While data augmentation was attempted to address the scarcity of positive examples, it did not lead to improved results, indicating that simply adding more data without careful consideration of its relevance or quality might not be effective.
  • "Text Shards" Potential: The "Text shards" approach, focusing on extracting and training on actual PCL segments, shows promise for improving multi-class classification by reducing noise, though its effectiveness was limited in this study.

Key Results

  • Subtask 1 (Binary Classification):
    • The best performance was achieved by BERT Transformers (specifically BertForSequenceClassification) with an F1 score of 0.5074.
  • Surprisingly, Logistic Regression applied to a stemmed, SMOTE-oversampled dataset was the second-best performer, achieving an F1 score of 0.38.
  • Subtask 2 (Multi-label Multi-class Classification):
    • Performance was generally low, with models struggling to learn all classes.
  • The best model, DistilBERT, achieved an F1 score of 0.34 on a multi-label classification task across 7 classes.
  • Training on 3 general classes first and then fine-tuning on 7 specific classes showed that learned features could be lost, and the best result for this approach was an F1 score of 0.26 with DistilBERT.

Limitations and Future Work

  • Limitations:
    • The significant class imbalance in the binary classification task limited overall performance.
  • The multi-label classification task proved very challenging, with models only learning a few classes.
  • The "general class" approach for multi-label classification was found to be inefficient.
  • Future Work:
    • Develop more sophisticated approaches that incorporate greater linguistic insight.
  • Explore more complex models and techniques that might not rely solely on specific pre-trained architectures.
  • Investigate better methods for handling the complexity of multi-label classification for PCL.

Summary updated

BibTeX
@inproceedings{dumitrascu2022university,
  author = {Tudor Dumitrascu and Raluca-Andreea Gînga and Bogdan Dobre and Bogdan Radu Silviu Sielecki},
  title = {{University of Bucharest Team at Semeval-2022 Task4: Detection and Classification of Patronizing and Condescending Language}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure