Tübingen at SemEval-2023 Task 4: What Can Stance Tell? A Computational Study on Detecting Human Values behind Arguments

Fidan CanACL 2023

TL;DR

This paper explores whether incorporating "stance" (whether a premise supports or opposes a conclusion) can improve the detection of underlying human values in arguments. The authors found that while their RoBERTa-based system achieved a respectable F1 score of 0.50 on the main test set, the performance dropped significantly on a more challenging, religiously-themed dataset, suggesting that implicit language and covert messages pose a substantial hurdle for current models.

What Problem Does This Paper Address?

Arguments are a fundamental part of human communication, used to express and rationalize ideas. However, the values that drive these arguments are often not explicitly stated, making them difficult to identify computationally. This paper tackles the challenge of "human value identification" – the task of determining which of 20 predefined human values are reflected in a given textual argument. The authors are particularly interested in whether understanding the stance of an argument's premise towards its conclusion can provide a crucial clue to uncovering these hidden values.

Methods

The core of the system utilizes a pre-trained language model, specifically RoBERTa-Large. The novel aspect of their approach is treating the stance of the premise towards the conclusion as an additional label to predict, alongside the 20 human values. This means the model is trained to output 21 labels in total. The input to the model is created by concatenating the "conclusion" and "premise" text fields from the dataset, separated by a special token.

For comparison and to investigate the impact of their choices, two other experimental setups were explored:

  1. Stance as Input: Stance information was included directly in the input text (concatenated with conclusion and premise) rather than as a separate output label.
  2. BERT-Large Model: The same task was performed using BERT-Large instead of RoBERTa-Large, to compare the effectiveness of different pre-trained models.

Datasets

The study uses two datasets for SemEval-2023 Task 4:

  • Main Dataset: This dataset was split into training, validation, and test sets. It comprises arguments with columns for "Argument ID," "Conclusion," "Stance," and "Premise." The human value labels were provided in separate files. This dataset is a compilation from various sources, including the IBM-ArgQ-Rank-30kArgs, Conference on the Future of Europe, and Group Discussion Ideas, covering multiple languages.
  • Nahj al-Balagha Dataset: This is a secondary test set consisting of 279 arguments extracted from Islamic religious texts. These texts are characterized by their eloquent and often implicit content, making them a more challenging benchmark for value identification. This dataset was manually translated into English.

Key Insights and Ideas

  • Stance as a Signal: The hypothesis that stance could improve human value identification was tested. While the main system incorporating stance as an output label performed reasonably well, the results suggest that stance alone might not be a silver bullet for all types of arguments.
  • Dataset Difficulty Matters: The significant performance drop on the Nahj al-Balagha dataset highlights the impact of dataset characteristics. Implicit language, covert messages, and eloquent content pose a much greater challenge than more straightforward arguments.
  • Model Architecture: The choice of transformer-encoder-only architectures (like BERT and RoBERTa) is justified for Natural Language Understanding (NLU) tasks, as they excel at comprehending text sequences.
  • RoBERTa vs. BERT: RoBERTa-Large, an optimized version of BERT, generally outperformed BERT-Large in the experimental setups, aligning with its reputation for improved performance.

Key Results

  • The main system, using RoBERTa-Large with stance as an additional predicted label, achieved an F1-score of 0.50 on the main test set, ranking ninth among participants.
  • On the secondary test set (Nahj al-Balagha), the same system achieved an F1-score of 0.35, indicating a substantial performance decrease.
  • The experimental setup where stance was included as part of the input text resulted in an F1-score of 0.45 on the validation set, which was lower than the main approach.
  • Using BERT-Large instead of RoBERTa-Large (with stance as a label) yielded an F1-score of 0.39 on the main validation set, also underperforming the main approach.
  • Certain human values, such as Stimulation and Humility, were consistently poorly predicted across both datasets, with very low F1-scores (e.g., 0.10 for Stimulation on the main set).

Limitations and Future Work

  • Data Frequency: The low frequency of certain human values in the datasets is acknowledged as a potential reason for poor prediction performance.
  • Implicit Language: The paper explicitly mentions that the implicit and covert nature of arguments in the Nahj al-Balagha dataset presents a significant challenge.
  • Future Directions:
    • Investigating the impact of language use and expression styles across different content types in a future comparative study with larger, more balanced datasets.
  • Exploring transfer-learning approaches, such as using models trained for ideology detection, to see if they can improve human value identification performance, especially for implicitly expressed values.
  • Training on larger and more diverse datasets might help models better predict human values embedded in covert arguments.

Summary updated

BibTeX
@inproceedings{can2023tubingen,
  author = {Fidan Can},
  title = {{Tübingen at SemEval-2023 Task 4: What Can Stance Tell? A Computational Study on Detecting Human Values behind Arguments}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure