Multi-Target User Stance Discovery on Reddit

Benjamin Steel, Derek RuthsACL 2024

TL;DR

This paper introduces a novel, end-to-end workflow for discovering users' stances on multiple topics simultaneously from social media data, addressing the challenge of comparing opinions across different subjects. Their method, applied to Reddit data, successfully reproduces known opinion trends and offers a new way to study nuanced political discourse.

What Problem Does This Paper Address?

Understanding public opinion is crucial for various fields, from political science to market research. Social media platforms offer a rich source of this data, but current methods for measuring user opinions often fall short. Practitioners need tools that can:

  1. Measure opinions on multiple topics at once: Existing research often focuses on a single topic or requires massive datasets for multi-topic analysis.
  2. Provide high temporal and topical resolution: Many methods operate at the post level, not the user level, or lack fine-grained temporal insights.
  3. Be practical and cost-effective: The ideal method should require minimal manual effort and data annotation.

A key challenge in multi-target stance detection is ensuring that opinion measurements are comparable across different topics, especially when the accuracy of stance detection varies by topic. This paper aims to bridge the gap between cutting-edge NLP research and the practical needs of opinion measurement.

Methods

The authors propose a three-stage pipeline:

  1. Stance Target Selection: This stage identifies relevant topics from the social media corpus. It uses language encoders (like sentence transformers) to embed text, dimensionality reduction (UMAP), and clustering (HDBSCAN, BERTopic) to discover thematic clusters. These clusters are then further refined to identify potential stance targets, which are manually selected by the practitioner based on domain expertise.
  2. Text Stance Detection: For each identified stance target, this stage classifies individual social media posts into "favor," "against," or "neutral." They build a small, annotated dataset (100 posts per target) and train stance classifiers using auto-regressive language models (specifically, Starling-7B). They explore parameter-efficient fine-tuning (PEFT) methods, finding that a "two-step PEFT" approach (fine-tuning on all stances, then on individual stances) yields the best results. An ensemble classifier is then used to combine predictions from the best individual classifiers for each stance target.
  3. User Stance Inference: This is the core novelty. To aggregate post-level stances into a user-level stance and ensure comparability across targets, they propose two methods:
  4. Weighted Mean: This method assigns weights to comment stances based on the precision of the classifier that made the prediction.
  5. Inferred Mean: This approach uses a probabilistic generative model to infer latent user stances, accounting for classifier error (precision and recall). This method aims to provide a more accurate characterization of variance.

Datasets

The authors utilized the Pushshift dataset to collect Reddit content from 2022. They focused on posts from four large Canadian-centric subreddits: r/canada, r/vancouver, r/ontario, and r/toronto. They created a custom dataset of 100 annotated posts per stance target for training and evaluating their text stance classifiers.

Key Insights and Ideas

  • Multi-target stance detection is inherently more complex: The varying accuracy of classifiers across different topics and labels necessitates careful aggregation and normalization to ensure comparability.
  • Assembling existing NLP tools can create powerful new workflows: The paper demonstrates that by thoughtfully combining techniques like topic modeling, LLM-based classification, and ensemble methods, a practical multi-target stance discovery system can be built.
  • Small, targeted annotation efforts can be surprisingly effective: The authors show that annotating just 100 posts per stance target is sufficient for training robust classifiers, making the process more accessible.
  • Inferred user stance provides a more nuanced view: The probabilistic "Inferred Mean" method, while computationally more intensive, offers a better characterization of user opinion variance compared to simpler weighted means, especially when classifier errors are present.
  • Political discourse on Reddit shows strong issue correlations: Users' stances on different political issues often align with each other, sometimes more strongly than with party affiliations, suggesting ideological sorting.

Key Results

  • The proposed ensemble classifier achieved an F1 score of 0.65 on average across stance targets, outperforming baselines.
  • The "two-step PEFT" method for training classifiers yielded the highest accuracy.
  • Analysis of Canadian political discourse on Reddit revealed strong correlations between stances on various issues (e.g., Liberals and Gun Control, Conservatives and Gun Control).
  • The temporal analysis showed that opinion shifts on topics like vaccine mandates align with real-world events and public sentiment changes.

Limitations and Future Work

The authors acknowledge several limitations:

  • Manual Stance Target Selection: The current method requires manual selection of stance targets, which can introduce bias. Future work could automate this process.
  • "Neutral" Stance Ambiguity: Defining and interpreting "neutral" stances can be challenging and topic-dependent.
  • User Stance Aggregation: While the "Inferred Mean" method is promising, it tends to underestimate user stance when classifier precision and recall are low. Further work on modeling user stance distributions (e.g., using beta distributions) is suggested.
  • Reproducibility: The method is not a "plug-and-play" solution and requires manual tuning. They have released their source code to facilitate further research.

Future work could focus on automating stance target discovery, extending the methodology to multi-modal platforms, and developing more sophisticated models to uncover complex opinion dynamics.

Summary updated

BibTeX
@inproceedings{steel2024multi,
  author = {Benjamin Steel and Derek Ruths},
  title = {{Multi-Target User Stance Discovery on Reddit}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure