Five Years of Argument Mining: a Data-driven Analysis

Elena Cabrio, Serena VillataIJCAI 2018 · DOI

TL;DR

This paper provides a comprehensive data-driven analysis of the argument mining (AM) field over its first five years (roughly 2014-2018). It maps out the landscape of AM tasks, common approaches, and datasets, highlighting the field's rapid growth and its interdisciplinary nature, while also identifying key challenges and future research directions.

What Problem Does This Paper Address?

Argument mining (AM) is an emerging field in Artificial Intelligence aiming to automatically identify and extract arguments and their relationships from text. The goal is to create machine-readable structured data that can power computational models of argument. While the concept has roots in formal argumentation theory and Natural Language Processing (NLP), the paper addresses the need for a structured overview of how this interdisciplinary field has evolved, particularly in its early, formative years. It seeks to understand the progress made, the common methodologies employed, the available resources, and the persistent challenges that hinder further advancement.

Methods

The paper adopts a "data-driven" perspective, analyzing existing research based on specific axes:

  • Application Scenarios: Categorizing AM research by the domain it's applied to (e.g., education, web-based content, legal documents, political debates, newspaper articles, social media).
  • Algorithms: Identifying the most frequently used machine learning and NLP techniques for AM tasks.
  • Features: Cataloging the types of linguistic and contextual features employed in AM models.
  • Datasets: Reviewing the available corpora used for training and evaluating AM systems.

The core AM framework is broken down into two main stages:

  1. Argument Extraction: Identifying individual arguments (claims and premises) within text.
  2. Relation Prediction: Determining the relationships (e.g., support, attack) between extracted arguments.

The paper also discusses the importance of inter-annotator agreement (using Cohen's kappa) for ensuring the reliability of annotated datasets.

Datasets

The paper lists and categorizes several datasets used in argument mining research. Key datasets mentioned include:

  • Persuasive Essays: Datasets for identifying argument components and structures in student essays.
  • Microtexts: Datasets for argument mining on short texts.
  • Wikipedia Articles: Datasets for claim detection and stance classification.
  • Web Debates/Forums: Corpora from online discussions (e.g., Debatepedia, 4forums.com) for argument extraction and relation prediction.
  • Legal Documents: Datasets from court judgments (e.g., ECHR) for identifying legal arguments.
  • Political Speeches/Debates: Corpora from parliamentary speeches and political debates.
  • Newspaper Articles: Datasets for argument mining in news content.
  • Social Media (Twitter): Datasets for identifying argumentative tweets.

The paper notes that while many datasets exist, they often have slightly different definitions and annotations, making direct comparison and unification challenging.

Key Insights and Ideas

  • Interdisciplinary Synergy: Argument mining emerged from the convergence of NLP and Knowledge Representation & Reasoning (KRR) communities, leveraging NLP's text processing capabilities and KRR's formal argumentation models.
  • Rapid Growth: The field experienced significant growth from 2014 onwards, marked by workshops, tutorials, and increasing involvement from companies like IBM.
  • Task Complexity: Argument mining is inherently complex, especially when dealing with "arguments in the wild" (natural language text) which deviate from idealized argumentation patterns. Relation prediction is often found to be more challenging than component detection.
  • Domain Specificity: While some approaches show promise across domains, argument mining often requires domain-specific adaptations due to variations in how arguments are expressed (e.g., legal cases vs. persuasive essays vs. tweets).
  • Data-Driven Focus: The paper emphasizes the importance of data-driven analysis and the creation of high-quality annotated corpora for advancing the field.
  • Connection to Broader AI: AM is linked to other critical AI areas like fact-checking, misinformation detection, and explainable AI, suggesting its potential impact beyond just argument analysis.

Key Results

The paper doesn't present new experimental results but rather synthesizes findings from existing work. Key observations include:

  • Algorithm Performance: Support Vector Machines (SVMs) were frequently the most performing algorithms for various AM sub-tasks in the early years. Recurrent Neural Networks (RNNs) also showed promise.
  • Feature Importance: Syntactic, positional, lexicon-based, topic-relatedness, sentiment, and discourse features were commonly used.
  • Task Difficulty: Relation prediction is generally considered more challenging than argument component detection.
  • Dataset Diversity: A wide range of datasets have been created, reflecting the diverse application scenarios and the evolving definitions within the field.

Limitations and Future Work

  • Limitations:
    • Dataset Heterogeneity: The lack of standardized definitions and annotations across datasets hinders direct comparison and generalization.
  • Human Agreement: Even human annotators struggle with consistency, indicating the inherent difficulty of AM tasks and setting a ceiling for automatic performance.
  • Multilinguality: Very few approaches tackled AM in languages other than English.
  • Future Work:
    • Improving System Performance: Continued efforts are needed to enhance the accuracy of AM systems, especially for challenging document types.
  • Bridging NLP and KRR: Addressing the gap between NLP's text processing and KRR's formal models remains a key challenge.
  • Generalization: Developing methods that can generalize across different text types and domains is crucial.
  • Multilingual AM: Expanding AM techniques to other languages.
  • New Challenges: Exploring areas like argument persuasion, fact-checking, misinformation detection, and cyberbullying prevention where AM can play a vital role.

Summary updated

BibTeX
@inproceedings{cabrio2018five,
  author = {Elena Cabrio and Serena Villata},
  title = {{Five Years of Argument Mining: a Data-driven Analysis}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/766},
}
Figure