Agreement and Disagreement: Comparison of Points of View in the Political Domain

Stefano Menini, Sara TonelliCOLING 2016

TL;DR

This paper tackles the challenging task of automatically detecting agreement and disagreement between politicians' viewpoints by moving beyond simple sentiment analysis. They propose a novel feature-rich approach that considers both the sentiment and the proposed solutions within political discourse, demonstrating its effectiveness on newly created datasets.

What Problem Does This Paper Address?

Comparing the viewpoints of politicians on specific topics is crucial for understanding political discourse, but it's a complex task. Traditional approaches often rely on classifying individual statements as simply "for" or "against" a topic, akin to sentiment analysis. However, this oversimplifies the nuances of political agreement and disagreement. For instance, two politicians might both oppose a regime but propose vastly different ways to address it, leading to disagreement despite shared negative sentiment. This paper aims to bridge this gap by developing a system that can automatically identify agreement and disagreement by considering multiple dimensions of a politician's stance, particularly in contexts where positions are not explicitly stated but inferred from broader document collections.

Methods

The authors propose a supervised machine learning approach, specifically using Support Vector Machines (SVM) with a radial kernel, to classify pairs of text snippets for agreement or disagreement. Their core innovation lies in the features they extract, which go beyond basic text analysis:

  • Sentiment Features: They leverage sentiment analysis modules to capture the sentiment expressed in each snippet, both globally and specifically within text segments related to the topic.
  • Semantic Features: To understand the proposed solutions or initiatives, they employ word embeddings (Word2vec) to represent topic keywords and calculate cosine similarity between these representations. They also use textual entailment to assess if one statement logically supports or contradicts another.
  • Surface Features: These include lexical overlap (common words) and negation cues, which can indicate subtle differences or similarities in how topics are discussed.

The system processes pairs of snippets, extracting these features at both the snippet level and a more focused "subtree" level (text directly related to the topic keywords).

Datasets

The paper introduces and utilizes three datasets:

  • 1960 Presidential Campaign Dataset: A pilot dataset created by manually annotating excerpts from Nixon and Kennedy's speeches related to 38 campaign topics. This dataset consists of 350 snippet pairs.
  • Extended 1960 Presidential Campaign Dataset: An augmented version of the pilot dataset, expanded to 1,400 snippet pairs and balanced for agreement/disagreement classes.
  • Debatepedia Dataset: A larger dataset extracted from Debatepedia, an online encyclopedia of debates. This dataset contains 29,354 snippet pairs, carefully curated to represent opposing sides of debates and organized by shared keywords as topics.

Key Insights and Ideas

  • Sentiment Alone Isn't Enough: The pilot study revealed that politicians can agree on a topic without sharing the same sentiment, and vice-versa. This highlights the need for features beyond simple positive/negative sentiment.
  • Solutions Matter: The proposed solutions or initiatives are a critical component in determining agreement or disagreement, even when sentiment is similar.
  • Implicit Agreement/Disagreement: In political discourse, agreement and disagreement are often not explicitly stated but inferred from a broader context. The system's ability to handle longer text snippets and extract nuanced features is key to capturing this.
  • Feature Combination is Crucial: No single feature category (sentiment, semantic, or surface) is sufficient on its own. Combining these different dimensions significantly improves classification performance.

Key Results

The proposed SVM-based classifier achieved good accuracy across all three datasets:

  • 1960 Elections Dataset: 83.0% accuracy with all features.
  • Extended 1960 Elections Dataset: 80.1% accuracy.
  • Debatepedia Dataset: 74.0% accuracy.

The results demonstrate that combining sentiment, semantic (entailment, word embeddings, cosine similarity), and surface features yields the best performance. Interestingly, adding coreference resolution features slightly decreased performance, suggesting potential issues with the coreference system itself or its interaction with the other features. The authors also showed that their core set of features, designed for pairwise comparison, is effective for classifying single statements as supporting or opposing a topic, achieving 87.2% accuracy on Debatepedia data.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Limited Dataset Size: The initial 1960 Elections dataset was small, impacting classification performance in some configurations.
  • Coreference Resolution Issues: The slight performance drop with coreference features warrants further investigation.
  • Neutral Judgments: The study focused on agreement and disagreement, excluding neutral judgments. Future work could incorporate this class for a more comprehensive analysis.
  • Broader Applications: The authors plan to integrate their agreement/disagreement detection module into a platform for analyzing political speeches and explore other machine learning models like Convolutional Neural Networks (CNNs). They also aim to investigate the performance gap observed with CNNs compared to SVMs.

Summary updated

BibTeX
@inproceedings{menini2016agreement,
  author = {Stefano Menini and Sara Tonelli},
  title = {{Agreement and Disagreement: Comparison of Points of View in the Political Domain}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
}
Figure