PiCkLe at SemEval-2022 Task 4: Boosting Pre-trained Language Models with Task Specific Metadata and Cost Sensitive Learning

Manan SuriNAACL 2022

TL;DR

This paper tackles the challenging task of detecting patronizing and condescending language (PCL) by fine-tuning pre-trained language models (PLMs). The authors demonstrate that incorporating task-specific metadata and employing cost-sensitive learning strategies significantly improves PCL detection, especially for underrepresented categories.

What Problem Does This Paper Address?

Patronizing and condescending language (PCL) is a subtle form of communication that, while seemingly helpful, often carries an underlying tone of superiority or pity towards vulnerable communities. Unlike overtly offensive language, PCL can be insidious, reinforcing stereotypes and masking deeper societal issues. Identifying PCL is crucial for fostering more equitable communication. SemEval-2022 Task 4 aimed to address this by introducing two subtasks: binary classification of PCL and multi-label classification of specific PCL categories. The primary challenge lies in the subtle nature of PCL and the significant class imbalance present in the datasets, where PCL instances and specific PCL categories are often underrepresented.

Methods

The core of the PiCkLe system relies on fine-tuning state-of-the-art pre-trained language models (PLMs) for the PCL detection task. The authors experimented with several popular PLMs, including BERT, RoBERTa, DistilBERT, and ALBERT. To enhance the models' understanding of context, they incorporated task-specific metadata (keywords related to vulnerable communities) by prepending them to the input text, separated by special tokens ([SEP] for BERT-like models and </s> for RoBERTa).

A significant challenge was the class imbalance. To address this, the researchers employed cost-sensitive learning techniques. This approach modifies the loss function to assign higher weights to misclassifications of minority classes, effectively increasing the "cost" of errors on underrepresented categories. They explored three weighting schemes: Inverse of Number of Samples (INS), Inverse of Square Root of Number of Samples (ISNS), and Effective Number of Samples (ENS).

Datasets

The study utilizes the "Don't Patronize Me!" dataset, which was used for SemEval-2020 Task 4. This dataset consists of news stories annotated for PCL and its specific categories. The authors used the train, development, and test splits provided by the task organizers. The dataset includes keywords related to vulnerable communities (e.g., "disabled," "homeless," "refugee") and is sourced from 20 countries.

Key Insights and Ideas

  • Metadata Matters: Prepending keywords related to vulnerable communities to the input text significantly improved model performance. This highlights how providing explicit contextual cues can help PLMs better understand the nuances of PCL.
  • Cost-Sensitive Learning is Crucial: The class imbalance in PCL datasets is a major hurdle. Cost-sensitive learning, particularly the ENS method, proved effective in boosting performance, especially for minority classes. This demonstrates that simply having more data isn't enough; the model needs to be incentivized to learn from the less frequent examples.
  • PLM Choice Influences Performance: While all tested PLMs benefited from the proposed techniques, BERT and RoBERTa generally showed stronger performance, suggesting their architectural strengths are well-suited for this task.
  • Subtlety of PCL: The paper underscores the inherent difficulty in detecting PCL due to its subtle nature, which often relies on implicit meanings and societal power dynamics rather than explicit offensive language.

Key Results

For Subtask 1 (Binary Classification), the best performing model was a finetuned BERT model with token-separated metadata and ENS class weighting, achieving an F1 score of 0.513 on the test set. This placed them 38th on the official leaderboard.

For Subtask 2 (Categorical Classification), the submitted model was a finetuned RoBERTa with ENS class weighting and token-separated metadata. While this model performed well on the development set (average F1 of 0.419), its performance on the evaluation set was lower (F1 of 0.1515), ranking them 35th. However, the analysis showed that this model performed particularly well on the 'the' class, which was heavily underrepresented in the training set, indicating the effectiveness of cost-sensitive learning.

Limitations and Future Work

The authors acknowledge that recognizing PCL is inherently difficult due to the subtle nature of the language used and the lack of an exact benchmark. They suggest that further tuning of parameters and improvements in paragraph representation could enhance performance.

For future work, they propose exploring other task-specific metadata utilization techniques and investigating other transformer-based models like T5 and ELECTRA. Additionally, they aim to improve the model's ability to recognize the subtle use of language that characterizes PCL. A key observation for Subtask 2's performance drop on the evaluation set was the potential bias from the development set's label distribution, suggesting a need for more robust generalization strategies.

Summary updated

BibTeX
@inproceedings{suri2022pickle,
  author = {Manan Suri},
  title = {{PiCkLe at SemEval-2022 Task 4: Boosting Pre-trained Language Models with Task Specific Metadata and Cost Sensitive Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure