"Tell me who you are and I tell you how you argue": Predicting Stances and Arguments for Stakeholder Groups

Philipp Heinisch, Lorik Dumani, Philipp Cimiano, Ralf SchenkelNAACL 2024 · DOI

TL;DR

This paper introduces ARGENST, a framework for predicting individual stakeholder stances and generating arguments on controversial topics based on their personal attributes. They propose two methods – a fine-tuned LLM and a prompting-based approach – and find that while prompting excels at stance prediction, the fine-tuned model generates more accurate arguments, though both have limitations in capturing nuanced opinions.

What Problem Does This Paper Address?

Understanding public opinion and the reasoning behind it is crucial for policy-making, conflict resolution, and fostering deliberation. However, existing argument mining research primarily focuses on identifying and extracting arguments from text, neglecting the prediction of who holds which stance and why. This paper addresses this gap by tackling the novel task of predicting both the stance (e.g., pro/con) and the argumentative "major claim" (the core reason) for individual stakeholders, given their personal attributes (demographics, political orientation, religion, etc.) and a controversial topic. This allows for a more granular understanding of stakeholder perspectives than simply aggregating opinions.

Methods

The authors propose ARGENST, a framework with two main instantiations:

  • Fine-tuned LLM Approach: This method uses a recommender-based architecture. It comprises two components: a stance classifier and a major claim generator. The stance classifier predicts a binary stance, and its internal representation is fed into a generative LLM (like T5) to produce the major claim. This approach also incorporates a "friendship graph" to capture user similarities.
  • Prompting Approach: This method leverages pre-trained Large Language Models (LLMs), specifically GPT-4 Turbo. It involves crafting prompts that include the task instructions, the topic, and the stakeholder's personal attributes. They experiment with zero-shot (no examples) and few-shot (with examples) settings.

Datasets

The primary dataset used is from debate.org, made available by Plenz et al. (2024). This dataset includes:

  • Threaded discussions on controversial topics.
  • User profiles with voluntary personal and demographic attributes (e.g., education level, income, political spectrum, religion).
  • Stances (votes) and arguments exchanged within topical threads.

The dataset was split into training, development, and testing sets, ensuring no overlap in topics or persons between splits.

Key Insights and Ideas

  • Individual-level prediction is key: Moving beyond group-level opinions to predict individual stances and arguments based on personal attributes offers a more nuanced understanding.
  • Prompting vs. Fine-tuning Trade-offs: There's a clear trade-off: prompting excels at accurately predicting stances, while fine-tuning is better at generating arguments that align with ground truth, as measured by BERTScore.
  • The danger of oversimplification: Both approaches, especially the prompting method with examples, can reinforce stereotypes and minimize diversity within groups, leading to a potentially misleading impression of homogeneity.
  • Friendship networks matter: For the fine-tuned approach, incorporating a friendship graph improves performance, suggesting that social connections influence argumentation.
  • Information sparsity impacts performance: The more personal information a user provides, the better the models (particularly prompting-based ones) perform, highlighting the importance of rich profiles.

Key Results

Automatic Evaluation: * Stance Prediction: The fine-tuned approach achieved modest accuracy (0.521), barely above random. GPT-4 (zero-shot) significantly outperformed it with 0.682 accuracy, and few-shot prompting further improved it to 0.698. * Major Claim Generation: The fine-tuned approach achieved a much higher BERTScore (0.644) compared to the prompting approach (0.157 for zero-shot, up to 0.344 for few-shot).

Manual Evaluation: * Stance Soundness: Annotators rated GPT-4 (3-shot coalition) as the most sound (2.93/4), followed by GPT-4 (zero-shot) (2.87/4), and then the fine-tuned approach (2.59/4). * Major Claim Quality: Annotators preferred the prompt-based approaches for major claims, finding them more plausible and precise than those from the fine-tuned model, which often generated vague or contradictory arguments.

Limitations and Future Work

Limitations: * Misrepresentation of Groups: The models might overemphasize majority views, leading to stigmatization and the enforcement of stereotypes. * Lack of Common Sense: The models may lack explicit common-sense knowledge and logical reasoning, leading to inconsistencies. * Data Bias: The dataset is primarily from English-speaking users in the US/UK, potentially biasing results towards Western culture. * Generalization to New Topics: The approach might struggle with topics not present in the training data. * Ethical Concerns: The potential for misrepresenting diverse opinions and creating a false sense of homogeneity within groups is a significant ethical risk. The authors stress that the model should not be used to predict individual opinions but rather group perspectives.

Future Work: * Retrieval-Augmented Architectures: Integrating retrieval mechanisms to access external knowledge could improve model understanding of topics and common sense. * Addressing Information Sparsity: Further research is needed to handle users with very limited profile information. * Improving Argument Generation: Developing methods to ensure generated arguments are not only accurate but also meaningful and consistent with the predicted stance.

Summary updated

BibTeX
@inproceedings{heinisch2024tell,
  author = {Philipp Heinisch and Lorik Dumani and Philipp Cimiano and Ralf Schenkel},
  title = {{"Tell me who you are and I tell you how you argue": Predicting Stances and Arguments for Stakeholder Groups}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.128},
}
Figure