Stance Prediction for Contemporary Issues: Data and Experiments

Marjan Hosseinia, Eduard Dragut, Arjun MukherjeeACL 2020

TL;DR

This paper introduces a novel dataset for stance detection on contemporary issues and proposes a shallow recurrent neural network model that leverages sentiment and emotion information. This approach achieves competitive results compared to larger transformer models like BERT, while using significantly fewer parameters and computational resources.

What Problem Does This Paper Address?

In today's digital landscape, online discussions on contemporary issues are often polarized and one-sided, contributing to confirmation bias and the spread of misinformation. Automatically identifying the stance (pro or con) of opinions towards a given issue is crucial for fostering critical thinking and providing balanced information. While pre-trained transformer models like BERT have shown promise, they are computationally expensive and require substantial fine-tuning. This paper aims to bridge this gap by developing a more efficient method for stance detection, particularly for longer, paragraph-like opinions commonly found in non-partisan resources.

Methods

The core of this work lies in two main areas: a novel dataset and a proposed model architecture.

Dataset: The authors created a new dataset by collecting 419 controversial issues and their associated pros and cons from procon.org. Each perspective in the dataset is a paragraph-length text (100-200 words) that supports its claim with arguments. This dataset is unique in its focus on longer opinions and its non-partisan origin.

Model Architecture: The proposed model, termed VADER-Sent-GRU, is a shallow recurrent neural network. It leverages pre-trained BERT embeddings as a base but does not fine-tune the entire BERT model. Instead, it incorporates auxiliary sentiment and emotion information.

  • Sentiment: A rule-based sentiment tool (VADER) is used to extract sentiment scores for each sentence, which are then embedded.
  • Emotion: An NRC emotion lexicon is used to capture eight basic emotions from words within the opinion.
  • Recurrent Network: These sentiment and emotion embeddings are concatenated with BERT token embeddings and fed into a bidirectional Gated Recurrent Unit (GRU).
  • Pooling: Max-pooling and average-pooling operations are applied to the GRU outputs to capture important information across the entire opinion, hypothesizing that tokens with higher engagement scores are more critical for stance prediction.
  • Classification: The pooled features are then passed through a fully connected layer for final stance classification.

This approach aims to achieve competitive performance with a significantly smaller model size and fewer parameters than fine-tuned BERT.

Datasets

  • Procon.org Dataset (New): This is the primary dataset introduced in the paper. It comprises 419 distinct controversial issues, each with corresponding pro and con perspectives. The dataset contains over 6,000 instances, with each perspective being 100-200 words long.
  • Existing Datasets (for context): The paper references other stance detection datasets, including those from SemEval 2016 (Task 6), Wikipedia claims (Bar-Haim et al., 2017), and other online debate resources, noting that their instances are often shorter sentences.

Key Insights and Ideas

  • Sentiment and Emotion are Powerful: Incorporating sentiment and emotion information, even with a shallow model, significantly boosts stance detection performance, sometimes outperforming larger models. This highlights the importance of affective cues in understanding opinions.
  • Efficiency Matters: The paper demonstrates that a much smaller, shallower model (VADER-Sent-GRU) can achieve comparable or even better results than heavily fine-tuned, large transformer models like BERT, especially in terms of recall and F1 score. This is crucial for real-world applications where computational resources and inference speed are critical.
  • Max-Pooling for Interpretability: The use of max-pooling provides a form of interpretability by highlighting which words or phrases contribute most to the model's stance prediction. This allows researchers to understand why a particular stance was predicted.
  • Dataset Quality for Long Opinions: The creation of a dataset with paragraph-length opinions is a valuable contribution, as most existing datasets focus on shorter text snippets. This better reflects the nature of opinions found in many online discussions and articles.

Key Results

  • VADER-Sent-GRU Performance: The VADER-Sent-GRU model achieved the highest recall and F1 score among the models tested with paired inputs, reaching 86.62% recall and 76.90% F1 score. This is a significant improvement over the baseline BERT model (75.37% recall, 75.92% F1).
  • Parameter Efficiency: The VADER-Sent-GRU model uses 20x fewer parameters than the BERT-base model, leading to faster training and lower resource requirements.
  • Emotion's Contribution: The NRC-Emotion-GRU model also showed strong performance, indicating that emotion information is beneficial for stance detection.
  • BERTCONS vs. BERT: While previous work showed BERTCONS (BERT with consistency loss) outperforming BERT, this study did not observe the same superiority, suggesting that the nature of the dataset (longer opinions) might influence the effectiveness of that specific approach.

Limitations and Future Work

  • Dataset Specificity: The authors acknowledge that their model's performance is evaluated on their newly created procon.org dataset. Future work could explore its generalization to other datasets with different characteristics.
  • Rule-Based Sentiment: The reliance on a rule-based sentiment tool (VADER) might limit its adaptability to nuanced or domain-specific language.
  • Interpretability Depth: While max-pooling offers some insight, more advanced interpretability techniques could be explored to further understand the model's decision-making process.
  • Future Directions: The authors suggest exploring the impact of sentiment and emotion more deeply, as well as investigating more sophisticated pooling mechanisms and attention strategies to further enhance stance detection performance.

Summary updated

BibTeX
@inproceedings{hosseinia2020stance,
  author = {Marjan Hosseinia and Eduard Dragut and Arjun Mukherjee},
  title = {{Stance Prediction for Contemporary Issues: Data and Experiments}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure