Epicurus at SemEval-2023 Task 4: Improving Prediction of Human Values behind Arguments by Leveraging Their Definitions

Christian Fang, Qixiang Fang, Dong NguyenACL 2023

TL;DR

This paper tackles the challenge of automatically identifying human values within arguments, a task crucial for understanding discourse and human behavior. The authors propose a novel approach that leverages detailed definitions of human values, derived from annotation instructions and survey questions, to improve the accuracy of NLP models. Their method significantly outperforms existing baselines, demonstrating the value of rich definitional context for this complex task.

What Problem Does This Paper Address?

Understanding the underlying human values that motivate arguments is a complex task with significant implications for fields ranging from social science research to political analysis and even AI safety. Human values, like "hedonism" (seeking pleasure) or "achievement" (striving for success), are subjective and can be interpreted differently. This subjectivity leads to challenges in creating consistent taxonomies and, consequently, in building reliable NLP models to detect them. Previous approaches often relied on simpler definitions or direct keyword matching, which can miss nuanced expressions of values. The SemEval-2023 Task 4 (ValueEval) aimed to address this by asking participants to classify arguments into one of 20 human value categories. This paper specifically addresses the gap in accurately capturing the multifaceted nature of human values within arguments, hypothesizing that richer, definitional information is key to improving model performance.

Methods

The core innovation of this work lies in how they incorporate human value definitions into their NLP models. They leverage two primary sources for these definitions: 1) the annotation instructions provided to human annotators for the ValueEval task, and 2) survey questions used in established human value taxonomies (like those from Schwartz and Rokeach). These definitions are transformed into a standardized format ("It is important to be/have...") to ensure consistency.

The problem is then framed as a Natural Language Inference (NLI) task. For each argument (premise), they pair it with each definitional statement for a given value category. A BERT-based model is trained to predict whether the premise "entails" the definitional statement. If the premise entails the definition, it suggests the argument aligns with that value. To arrive at a single prediction per argument and value category, they average the entailment probabilities across all definitional statements for that value and then apply a learned threshold. This averaging strategy is inspired by social science measurement techniques, aiming to improve reliability by aggregating multiple indicators.

Datasets

The primary dataset used is the SemEval-2023 Task 4 (ValueEval) dataset, comprising 8,865 annotated arguments. This dataset was split into training (61%), validation (21%), and testing (18%) sets. A secondary test set, the "Nahj al-Balagha" dataset, containing 279 arguments from Islamic religious texts, was also used to evaluate the models' generalization capabilities on out-of-distribution data. The paper also references the underlying survey instruments (PVQ5X, Schwartz Value Survey, World Value Survey, Rokeach Value Survey, Life Values Inventory) from which they derived their definitional statements.

Key Insights and Ideas

  • Definitions are Powerful: The central finding is that providing NLP models with rich, human-readable definitions of abstract concepts like human values significantly boosts performance. This moves beyond simple keyword matching or shallow embeddings.
  • NLI as a Framework: Framing value identification as a Natural Language Inference problem proves effective, allowing models to reason about the relationship between an argument's premise and a value's definition.
  • Averaging for Robustness: Mimicking social science measurement by averaging predictions across multiple definitional statements for a single value category enhances model reliability and performance, especially on out-of-distribution data.
  • Annotation Instructions are Key: For in-domain data, using the actual annotation instructions provided to human annotators yielded better results than using more general survey questions, suggesting that these instructions capture the specific nuances relevant to the task.
  • Scalability with Fewer Definitions: Surprisingly, using only a small subset of definitional statements (as few as two or three per category) could achieve performance comparable to using all available definitions, drastically reducing computational overhead.

Key Results

The proposed models consistently outperformed the challenge organizers' baselines on the main test set. The best-performing model, using annotation instructions and weighted cross-entropy loss (ANNw), achieved a macro F1 score of .45, a 15% improvement over the BERT baseline. On the religious texts test set, which represents an out-of-distribution scenario, one of their models (SVYuw) achieved an 18% higher F1 score than the BERT baseline, highlighting the generalization benefits of their approach. The paper also notes that the inter-rater agreement on the dataset was relatively low (Krippendorff's α ≈ 0.49 for level-1 values, ≈ 0.31 for level-2 values), suggesting that the task itself is inherently challenging due to the subjective nature of values and potentially ambiguous annotation guidelines.

Limitations and Future Work

The authors acknowledge that their study was focused on testing the core idea of using definitional statements, rather than achieving state-of-the-art performance. They did not explore more advanced techniques like paraphrasing or ensemble learning, nor did they extensively fine-tune hyperparameters. A significant limitation highlighted is the low inter-rater agreement observed in the dataset, which suggests that improvements in the human value taxonomy or the annotation scheme itself might be necessary for even more reliable measurements. Future work could involve exploring larger language models, more sophisticated NLI techniques, and investigating how to further optimize the selection of definitional statements for improved efficiency and performance. The scalability finding also opens avenues for more efficient training pipelines.

Summary updated

BibTeX
@inproceedings{fang2023epicurus,
  author = {Christian Fang and Qixiang Fang and Dong Nguyen},
  title = {{Epicurus at SemEval-2023 Task 4: Improving Prediction of Human Values behind Arguments by Leveraging Their Definitions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure