TL;DR
This paper introduces ValueEval'23, a shared task and dataset for automatically identifying human values embedded within arguments. The goal is to understand the implicit justifications behind persuasive language, a crucial step for more nuanced AI in areas like argument analysis and ethical reasoning. The results show promising progress with transformer models, but also highlight significant challenges in detecting certain values.
What Problem Does This Paper Address?
People often disagree because they prioritize different underlying human values, even when presented with the same information. These values, like "self-direction" or "benevolence," act as implicit justifications for why certain actions or beliefs are considered desirable. Automatically identifying these values in natural language arguments is a complex but important problem. It can help AI systems understand the deeper ethical reasoning behind arguments, leading to better argument analysis, generation, and even more morally aligned AI interactions. Existing work has explored value taxonomies and some computational approaches, but a dedicated task and comprehensive dataset for this specific problem were needed.
Methods
The core of this work is the ValueEval'23 shared task, which framed the problem as a multi-label classification task. Given an argument (composed of a premise and conclusion) and a human value category, systems had to predict whether the argument appealed to that value. The task utilized a taxonomy of 20 value categories, derived from established social science research (Schwartz, 1994). Participants primarily employed transformer-based models, with many using ensemble techniques to combine predictions from multiple models. Some approaches explored non-classification methods like natural language inference and question-answering.
Datasets
The primary dataset for ValueEval'23, called Touché23-ValueEval, comprises over 9,000 arguments collected from diverse sources including religious texts, political discussions, newspaper editorials, and online platforms. This dataset is split into a main set (8865 arguments) for training, validation, and testing, and a supplementary set (459 arguments) designed to test out-of-domain generalizability. The supplementary set includes arguments from sources like Zhihu, Nahj al-Balagha, and The New York Times, presenting different genres and argument structures.
Key Insights and Ideas
- Implicit Values are Key: The success of arguments often hinges on unstated appeals to human values, making their identification a critical aspect of understanding persuasion.
- Transformer Dominance: Transformer models, particularly when ensembled, proved to be the most effective approach for this task, significantly outperforming simpler baselines.
- Domain Matters: Performance varied considerably between the in-domain main dataset and the out-of-domain supplementary datasets, highlighting the challenge of generalization.
- Value Difficulty Varies: Some value categories, like "nature" and "universalism: concern," were identified with relatively high accuracy, while others, such as "stimulation" and "hedonism," proved much more challenging.
- Ensembling Pays Off: Combining predictions from multiple models, especially those trained on different folds or with different objectives, consistently improved performance.
Key Results
The ValueEval'23 task saw 39 teams submit approaches, with the top performers achieving macro F1-scores of up to 0.87 for individual categories on the main test set. The best overall macro F1-score on the main dataset was 0.56, a substantial improvement over the BERT baseline (0.42). For the supplementary datasets, performance was lower but still encouraging, with teams demonstrating generalization capabilities despite limited or no in-domain training data. For instance, the best F1-score on the Nahj al-Balagha dataset reached 0.40, and on The New York Times dataset, 0.34. However, certain value categories, like stimulation, hedonism, face, and humility, saw much lower peak F1-scores, around 0.39, indicating areas ripe for future research.
Limitations and Future Work
The paper acknowledges that despite promising results, many challenges remain, particularly in identifying values that are implicitly expressed or less frequently used. The significant performance gap between the main and supplementary datasets underscores the need for more robust models that can generalize across different domains and argument styles. Future work could focus on developing models that better leverage value descriptions, explore more sophisticated ensemble methods, and potentially create larger, more diverse datasets to further improve generalization. The authors also suggest that the supplementary validation set could be beneficial for training future models.
Summary updated