ASRtrans at SemEval-2022 Task 4: Ensemble of Tuned Transformer-based Models for PCL Detection

Ailneni Rakshitha RaoNAACL 2022

TL;DR

This paper tackles the challenging task of automatically detecting patronizing and condescending language (PCL) in text, particularly towards vulnerable communities. The authors propose an ensemble of fine-tuned transformer models, including RoBERTa, XLNet, and Ernie-2.0, to achieve state-of-the-art performance on this nuanced NLP problem, outperforming baselines and ranking highly in the SemEval-2022 competition.

What Problem Does This Paper Address?

Patronizing and condescending language (PCL) is a subtle form of bullying that can contribute to inequalities and exclusion, especially when directed at vulnerable groups. Detecting PCL is difficult due to its subjective nature, reliance on world knowledge, and the need to distinguish it from genuine descriptions of difficult situations. SemEval-2022 Task 4 specifically aimed to address this by introducing a dataset for PCL detection, presenting two sub-tasks: binary classification of PCL (Task A) and multi-label classification into specific PCL categories (Task B). The key challenges highlighted are extreme class imbalance within the dataset and the inherent subtlety of PCL, making it hard even for humans to identify consistently.

Methods

The core of the proposed solution lies in leveraging and adapting powerful pre-trained transformer-based language models. The authors experimented with several models, including RoBERTa, XLNet, Ernie-2.0, and BERT-large. A key innovation is the use of a "tuned RoBERTa" (tRoBERTa), which was further trained on external datasets related to metaphor and condescension to improve its contextual embeddings for this specific task.

For Task B, which involves multi-label classification, they explored different problem transformation techniques:

  • Binary Relevance: Treats each label independently.
  • Label Powerset: Considers each unique combination of labels as a single class.
  • Classifier Chains: Chains binary classifiers, where the output of one classifier informs the next, allowing for the modeling of label dependencies.

The final system is an ensemble of these tuned models. They specifically employed a weighted-average ensemble, where the predictions of individual models are combined using learned weights. This approach aims to capture diverse linguistic nuances that different models might focus on. They also investigated multi-task learning and single-task learning setups.

Datasets

The primary dataset used is the "Don’t Patronize Me!" dataset (Pérez-Almendros et al., 2020), which was provided for the SemEval-2022 Task 4. This dataset consists of paragraphs extracted from the News on Web (NoW) corpus, covering texts from 20 English-speaking countries related to vulnerable communities.

For pre-training and fine-tuning, the authors also utilized external data:

  • TalkDown corpus (Wang and Potts, 2019)
  • MOH corpus (Mohammad et al., 2016)
  • VUA dataset (Steen et al., 2010)
  • Trofi dataset (Birke and Sarkar, 2006)

Key Insights and Ideas

  • Tuning Pre-trained Models is Crucial: Simply using off-the-shelf transformer models wasn't enough. Further training on domain-specific (metaphor, condescension) and related datasets significantly improved performance, highlighting the importance of domain adaptation.
  • Ensembling is Powerful: Combining predictions from multiple diverse transformer models (tRoBERTa, XLNet, Ernie-2.0) through weighted averaging yielded better results than individual models, demonstrating the benefit of leveraging complementary strengths.
  • Class Imbalance is a Major Hurdle: The extreme class imbalance in the dataset significantly impacted performance, especially for minority classes. Techniques like focal loss were employed to mitigate this.
  • Subtlety Requires World Knowledge: Classes requiring more world knowledge (e.g., "Shallow solution," "Presupposition") were harder to detect, suggesting that current models still struggle with deep semantic understanding and commonsense reasoning.
  • Classifier Chains Show Promise for Multi-label Tasks: While not the top performer in their experiments, Classifier Chains demonstrated better performance than Binary Relevance by accounting for label dependencies, a crucial aspect in multi-label classification.

Key Results

The authors achieved strong results in the SemEval-2022 Task 4 competition:

  • Task A: Ranked 52nd out of 81 teams with an F1 score of 0.4421.
  • Task B: Ranked 33rd out of 81 teams with an F1 score of 0.1889.

Their ensemble model significantly improved upon the baseline RoBERTa model. For instance, on Task A (Dev set), their ensemble achieved an F1 score of 0.543, compared to the baseline RoBERTa's 0.521. On Task B (Dev set), their ensemble achieved an F1 score of 0.1904, outperforming the baseline RoBERTa's 0.1041. They also noted that the "poorer the merrier" class saw a significant F1 score improvement, likely due to the tuned RoBERTa.

Limitations and Future Work

  • Limitations:
    • The inherent subtlety of PCL and the need for world knowledge remain significant challenges, leading to lower performance on certain classes.
  • The dataset, while valuable, is imbalanced, posing difficulties for model training.
  • While data augmentation helped, the authors noted that some techniques (like EDA) could degrade performance due to contextual mismatches.
  • Future Work:
    • Explore more sophisticated Classifier Chain strategies to better model label dependence and hierarchy.
  • Investigate other forms of data augmentation and techniques to address class imbalance.
  • Further research into incorporating commonsense reasoning and world knowledge into NLP models for tasks like PCL detection.

Summary updated

BibTeX
@inproceedings{rao2022asrtrans,
  author = {Ailneni Rakshitha Rao},
  title = {{ASRtrans at SemEval-2022 Task 4: Ensemble of Tuned Transformer-based Models for PCL Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure