Stance Classification, Outcome Prediction, and Impact Assessment: NLP Tasks for Studying Group Decision-Making

Elijah Mayfield, Alan Black

TL;DR

This paper introduces three novel NLP tasks – stance classification, outcome prediction, and individual impact assessment – to analyze group decision-making from text. They release a large corpus of Wikipedia "Articles for Deletion" debates, demonstrating that advanced language models like BERT significantly outperform traditional methods in these tasks, offering new tools for social science research.

What Problem Does This Paper Address?

Studying how groups make decisions is crucial for understanding teamwork and decision quality. However, analyzing the rich, unstructured text of group discussions is computationally challenging. Traditional social science methods often rely on simplified proxies or controlled experiments, limiting the ability to study real-world, complex group interactions. This paper aims to bridge this gap by developing NLP techniques that can extract meaningful insights from discussion transcripts, enabling a deeper understanding of group dynamics, conflict resolution, and the factors influencing decision outcomes.

Methods

The authors propose three NLP tasks:

  • Stance Classification: A supervised task to identify an individual's stance (e.g., "Keep" or "Delete" an article) based solely on their rationale text, masking explicit vote labels.
  • Outcome Prediction: A distantly supervised task where the model predicts the final outcome of a discussion (e.g., whether an article is kept or deleted) using only the rationale texts of all contributions, with the final outcome label serving as the sole supervision signal. This reduces the need for extensive manual annotation.
  • Individual Impact Assessment: An unsupervised extension of outcome prediction that quantifies the influence of individual contributions or users on the discussion's trajectory. This is achieved by measuring how much a contribution shifts the predicted outcome of the debate.

They evaluate these tasks using various language representations, including traditional bag-of-words, GloVe embeddings, and the state-of-the-art BERT contextual embeddings.

Datasets

The paper introduces a new, large-scale corpus of over 423,000 Wikipedia "Articles for Deletion" (AfD) debates, collected from January 1, 2005, to December 31, 2018. This corpus is preprocessed and released under an open-source license. AfD discussions are a natural fit for studying group decision-making as they involve nominations, votes, and comments leading to a consensus-based outcome determined by administrators.

Key Insights and Ideas

  • BERT's Superiority: Contextualized word embeddings, particularly BERT, consistently outperform traditional bag-of-words and static embeddings like GloVe across all proposed tasks. This highlights the power of modern NLP models in capturing nuanced meaning in conversational text.
  • Tractability of NLP Tasks: The research demonstrates that NLP can effectively tackle complex tasks like predicting stances and outcomes from unstructured discussion text, even with limited or distant supervision.
  • Impact Beyond Stance: The "Individual Impact Assessment" task offers a novel way to quantify influence, moving beyond simple vote counts or explicit stances to understand how specific contributions shape the discussion's direction. This has significant implications for understanding power dynamics and participation in online communities.
  • Wikipedia as a Rich Testbed: AfD discussions provide a valuable, real-world dataset for studying group decision-making, offering a balance of structured debate and natural language expression.

Key Results

  • Stance Classification: BERT achieved an accuracy of 82.0% and a Cohen's κ of 0.60, significantly outperforming GloVe (81.8% accuracy, 0.59 κ) and bag-of-words (76.0% accuracy, 0.45 κ) when using only rationale text.
  • Outcome Prediction: For full discussions, BERT achieved 85.8% accuracy and 0.62 κ, outperforming GloVe (81.7% accuracy, 0.49 κ) and bag-of-words (84.2% accuracy, 0.58 κ). In real-time predictions, BERT also maintained the highest accuracy.
  • Individual Impact: The impact metric, derived from BERT's outcome prediction, showed promise in identifying influential contributions and users, aligning with qualitative intuitions and revealing patterns like the disproportionate influence of early "Keep" votes in certain contexts.

Limitations and Future Work

  • Stance Classification Limitations: The authors acknowledge that some stance classification errors occur when contributions lack sufficient content or express user uncertainty. Future work could involve multi-task learning to incorporate uncertainty detection.
  • Outcome Prediction for "Keep": BERT's performance in predicting "Keep" outcomes, especially in short discussions, is notably weaker than predicting "Delete." This suggests an opportunity to better identify persuasive early "Keep" votes.
  • Impact Validation: The impact assessment metric is still preliminary and requires further validation through human annotators and downstream tasks.
  • Distant Supervision for Comments: The current stance classification excludes non-voting comments. Future work could explore distant supervision to label these comments.
  • Resource Intensity: While BERT shows superior performance, its computational demands might be a barrier for some social science researchers. The paper notes that a 20x increase in training data only yielded a <1% accuracy improvement, suggesting that for some tasks, smaller models might suffice.

Summary updated

Abstract

In group decision-making, the nuanced process of conflict and resolution that leads to consensus formation is closely tied to the quality of decisions made. Behavioral scientists rarely have rich access to process variables, though, as unstructured discussion transcripts are difficult to analyze. Here, we define ways for NLP researchers to contribute to the study of groups and teams. We introduce three tasks alongside a large new corpus of over 400,000 group debates on Wikipedia. We describe the tasks and their importance, then provide baselines showing that BERT contextualized word embeddings consistently outperform other language representations.
BibTeX
@inproceedings{mayfield2019stance,
  author = {Elijah Mayfield and Alan Black},
  title = {{Stance Classification, Outcome Prediction, and Impact Assessment: NLP Tasks for Studying Group Decision-Making}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}

Loading related content...

Figure