T.M. Scanlon at SemEval-2023 Task 4: Leveraging Pretrained Language Models for Human Value Argument Mining with Contrastive Learning

Milad Molazadeh Oskuee, Mostafa Rahgouy, Hamed Babaei Giglou, Cheryl D SealsACL 2023

TL;DR

This paper tackles the challenge of automatically identifying human values within arguments, a crucial step for understanding societal conflicts and improving communication. They leverage contrastive learning with DeBERTa-Large to enhance the isotropy of language model representations, leading to improved performance in this complex task.

What Problem Does This Paper Address?

Understanding human values is fundamental to social sciences, as differing priorities often fuel conflicts and shape individual and cultural behaviors. Automatically identifying these values within textual arguments is a significant challenge. This is important for applications like faceted search, generating persuasive arguments, and even profiling individuals based on their underlying values. The SemEval-2023 Task 4 (ValueEval) specifically aims to address this by classifying whether a given argument draws upon one of 20 predefined human value categories. Existing methods often struggle to capture the nuanced semantic relationships required for this task.

Methods

The core of this work lies in fine-tuning a pre-trained language model, specifically DeBERTa-Large, for the multi-label classification task of human value argument mining. To enhance the model's ability to discern subtle differences between arguments, they employ a contrastive learning (CL) approach. This CL regularizer aims to make the model's internal representations more "isotropic" – meaning they are more uniformly distributed in the feature space. This isotropy is believed to improve the model's ability to capture semantic information and reduce noise.

Specifically, they concatenate the premise, stance, and conclusion of an argument as input to the DeBERTa model. For fine-tuning, they freeze the initial layers of DeBERTa-Large, allowing the later layers to adapt to the specific task while retaining the general knowledge from pre-training. The model uses a sigmoid classifier to output probabilities for each of the 20 value categories. The loss function combines a standard binary cross-entropy (BCE) loss with a contrastive learning loss (CLreg), weighted by a hyperparameter $\lambda$. The CLreg component encourages similar representations for augmented versions of the same input while pushing dissimilar representations from different inputs further apart.

Datasets

The primary dataset used is the Touché23-ValueEval dataset. This dataset consists of argumentative texts, each labeled with one or more human values. The arguments are provided with premise text, conclusion text, and a binary stance (in favor of or against). The paper focuses on the "Main" dataset, which comprises 8865 arguments and 595 unique conclusions, sourced from three different collections between 2019 and 2022.

Key Insights and Ideas

  • Isotropy Matters: The paper strongly suggests that achieving isotropy in the feature space of pre-trained language models is crucial for improving performance on tasks like human value argument mining. This means that the way language models represent meaning can be improved by ensuring a more uniform distribution of these representations.
  • Contrastive Learning as a Regularizer: Contrastive learning isn't just for learning representations from scratch; it can be effectively used as a regularizer during fine-tuning to enhance existing models. By forcing the model to distinguish between similar and dissimilar arguments, it learns more robust and discriminative features.
  • DeBERTa-Large's Potential: The DeBERTa-Large model, with its substantial number of parameters, shows promise for this task, especially when combined with contrastive learning.
  • Ensemble Learning's Role: The results indicate that while a single model can perform well on certain value categories, achieving top performance across all categories might require ensemble learning approaches, as no single model excels everywhere.

Key Results

The proposed model achieved an F1 score of 49%, securing 11th place among 39 participants in the SemEval-2023 Task 4. Notably, their model ranked 1st in the Hedonism class, outperforming even the early bird submission. The model also demonstrated a significant improvement over baseline models, particularly the "1-baseline." The paper highlights that Large Language Models (LLMs) with more parameters are generally beneficial, but isotropy further enhances their performance.

Limitations and Future Work

The authors acknowledge that while their model performed well, the data augmentation technique might not have fully solved the task, as LLMs can still suffer from semantic overlaps in identifying human values. They also note that their model's performance was similar to the BERT baseline for many classes, suggesting that while effective, it might not be a universal solution for all value categories. Future work could explore ensemble learning approaches to further boost performance across all value categories. The paper also implies that further research into isotropization techniques for human value detection is warranted.

Summary updated

BibTeX
@inproceedings{oskuee2023t,
  author = {Milad Molazadeh Oskuee and Mostafa Rahgouy and Hamed Babaei Giglou and Cheryl D Seals},
  title = {{T.M. Scanlon at SemEval-2023 Task 4: Leveraging Pretrained Language Models for Human Value Argument Mining with Contrastive Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure