Using Summarization to Discover Argument Facets in Online Idealogical Dialog

Amita Misra, Pranav Anand, Jean E. Fox Tree, Marilyn WalkerNAACL 2015

TL;DR

This paper introduces a novel method for automatically identifying the core arguments (facets) within online ideological discussions, like those about gay marriage. By leveraging human-generated summaries and a new task called Argument Facet Similarity (AFS), they achieve a significant improvement over existing methods in grouping similar arguments across different conversations.

What Problem Does This Paper Address?

The internet is awash with dialogic, opinionated content, particularly in online forums discussing social and political issues. While previous work has focused on understanding the structure of these discussions (e.g., agreement/disagreement, stance), this paper tackles a more nuanced problem: identifying the recurring, abstract arguments or "facets" that underpin these debates. The challenge lies in recognizing that different specific propositions can represent the same underlying argumentative point, even across different conversations. For example, arguments about "legal protections" and "civil benefits" in the context of gay marriage both point to the same facet: the desire for the rights and benefits associated with marriage.

Methods

The authors propose a two-stage approach:

  1. Central Proposition Detection: They hypothesize that the most salient arguments within a dialog are those that people find most important, and that this salience is reflected in human summarization behavior. They use the "Pyramid method," a technique for evaluating summary quality, to identify these "central propositions" (CPs) from human summaries. CPs are essentially the core argumentative points of a single dialog.
  2. Argument Facet Induction: Once CPs are extracted from multiple dialogs on the same topic, the next step is to group them into broader "facets." To do this, they introduce a new task called Argument Facet Similarity (AFS). This task aims to measure the similarity between two CPs, not just on their surface-level semantic similarity, but on whether they represent the same underlying argumentative point (facet), even if the stances are contradictory. They then use clustering on these CPs to group them into facets.

Datasets

The primary dataset used in this work is derived from the Internet Argument Corpus (IAC). From this corpus, they selected 85 dialogs on the topic of gay marriage, specifically choosing dialogs with at least two different perspectives, multiple turns per contributor, and posts under 250 words.

They also created a new benchmark dataset for the AFS task by:

  • Collecting human summaries of these dialogs.
  • Using the Pyramid method to identify Central Propositions (CPs) from these summaries.
  • Clustering these CPs to generate pairs of arguments.
  • Having Mechanical Turk workers annotate the similarity of these argument pairs on a 6-point scale, specifically designed to capture facet similarity.

Key Insights and Ideas

  • Summaries as a Probe for Core Arguments: The core idea that human summaries can effectively reveal the most important argumentative points (Central Propositions) is a key insight. This leverages the natural tendency of humans to distill essential information.
  • Argument Facet Similarity (AFS) as a Distinct Task: The paper argues that AFS is different from standard Semantic Textual Similarity (STS). AFS focuses on whether arguments address the same underlying issue or "facet," regardless of specific wording or even stance, which is crucial for understanding the broader landscape of an argument.
  • Bottom-Up Discovery of Facets: The authors emphasize that they do not pre-define argument facets. Instead, they discover them through clustering, allowing for a more organic and data-driven understanding of how arguments are structured in online discussions.
  • Complementarity of Features: The results suggest that combining different types of features (like N-grams, LIWC, ROUGE, and DISCO) leads to better performance than relying on any single feature set, highlighting the multifaceted nature of argument similarity.

Key Results

The paper introduces a novel task, AFS, and demonstrates that their proposed method can predict AFS with a correlation of .54. This significantly outperforms:

  • An NGRAM overlap baseline: .39
  • A Semantic Textual Similarity (STS) system (UMBC): .45

When combining their proposed features with UMBC, they achieved the highest correlation of .54. They also found that LIWC and ROUGE features complement Ngram features, leading to significant improvements.

Limitations and Future Work

  • Topic Specificity: While the authors believe their trained system for AFS might generalize to other domains without retraining due to the topic-agnostic nature of their features, this has not yet been empirically demonstrated.
  • Human Summarization Dependency: The current method relies on human summaries to identify Central Propositions. Future work aims to automatically identify CPs directly from the source text, bypassing the need for human summarizers.
  • Expanding to More Topics: The current study is focused on gay marriage. Future work plans to expand the analysis to more topics to test the generalizability of their approach.
  • Improving Feature Sets: The authors suggest exploring other off-the-shelf systems, such as those for sentence specificity, to potentially enhance their feature sets.

Summary updated

BibTeX
@inproceedings{misra2015using,
  author = {Amita Misra and Pranav Anand and Jean E. Fox Tree and Marilyn Walker},
  title = {{Using Summarization to Discover Argument Facets in Online Idealogical Dialog}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure