Fallacious Argument Classification in Political Debates

Pierpaolo Goffredo, Shohreh Haddadan, Vorakit Vorakitphan, Elena Cabrio, Serena VillataIJCAI 2022 · DOI

TL;DR

This paper tackles the challenging task of automatically identifying fallacious arguments in political debates. The authors introduce a novel, large-scale annotated dataset of US presidential debates and propose a transformer-based neural architecture that leverages argumentation features to achieve state-of-the-art results in classifying these fallacies.

What Problem Does This Paper Address?

Political discourse is rife with fallacious arguments – flawed reasoning used to persuade or mislead. While identifying these fallacies is crucial for critical thinking and combating misinformation, automatically detecting them, especially within the complex context of political debates, remains a significant challenge. Existing work has often focused on specific fallacy types or lacked comprehensive datasets. This paper aims to bridge this gap by creating a rich resource and developing a robust method for classifying various fallacies in political debates, considering not just the fallacy itself but also its argumentative components and relations.

Methods

The core of the paper's methodology involves two main contributions:

  1. Dataset Creation: The authors extend the existing ElecDeb60To16 dataset, which contains US presidential debates from 1960-2016. They meticulously annotate six main categories of fallacious arguments (Ad Hominem, Appeal to Emotion, Appeal to Authority, Slippery Slope, False Cause, Slogan) and their sub-categories. This results in a dataset of 1628 annotated fallacious arguments, making it the largest of its kind for political debates.
  2. Neural Architecture: To classify these fallacies, the researchers propose a novel transformer-based neural architecture. They leverage advanced pre-trained language models (PLMs) like Longformer, which are capable of handling long input sequences, crucial for analyzing the context of debates. A key innovation is the integration of "argumentation features," specifically argument components (premise, claim) and relations (support, attack), into the classification process. This is achieved by incorporating separate loss functions for these features during training, allowing the model to learn from the structural aspects of arguments alongside their textual content.

Datasets

  • ElecDeb60To16 dataset: This is the primary dataset used. The authors extend its annotations to include fallacious arguments.
  • Newly Introduced Benchmark: The paper introduces a novel, large-scale annotated resource of 31 political debates from US Presidential Campaigns, featuring 1628 annotated fallacious arguments across six main categories and 14 sub-categories. This dataset is made publicly available.

Key Insights and Ideas

  • Argument Structure Matters: The most significant insight is that incorporating explicit argumentation features (components and relations) substantially improves the performance of fallacy classification models. This suggests that understanding how an argument is structured is as important as understanding what it says.
  • Long-Context Models are Essential: Political debates are lengthy and nuanced. Standard transformer models with fixed, short sequence lengths struggle to capture the full context. The use of models like Longformer, designed for long documents, is critical for effective analysis.
  • Fallacy Distribution Varies: The analysis of the dataset reveals interesting trends, such as the increased use of "Ad Hominem" in the 2016 debates and a higher prevalence of "Appeal to Fear" in the 2004 debates, potentially linked to the political climate and dominant topics of those years.
  • Subtle Fallacies are Challenging: The error analysis highlights that fallacies aiming to manipulate sentiment (like Slogans, Appeal to Emotions, and Ad Hominem) are particularly difficult to classify, often sharing vocabulary and relying on empathy, making them hard to distinguish from genuine arguments or even from each other.

Key Results

  • The proposed Longformer-based architecture, enhanced with argumentation features and a joint loss function, achieved the highest F1-score for fallacious argument classification, outperforming standard baselines and previous approaches.
  • Specifically, when using argumentation features (components and relations), the model achieved a macro-average F1-score of 0.84 for main fallacy categories, a significant improvement over the baseline without these features (0.57).
  • The dataset analysis showed "Appeal to Emotion" as the most frequent fallacy (1016 instances), while "Slippery Slope" was the least frequent (57 instances).
  • The paper also reports on the classification of 14 sub-categories, achieving promising results on some of the more frequent ones like "Flag waving" and "Slogan."

Limitations and Future Work

  • Annotation Nuance: The authors acknowledge that some fallacies are subtle and can be hard to pin down to a single category. They suggest that assigning multiple fallacy categories to an argument could be a future direction.
  • External Knowledge: Some classification errors require external knowledge (e.g., archival articles) to be correctly resolved, indicating a limitation of relying solely on the debate text and its argumentative structure.
  • Counter-Argument Generation: The paper expresses interest in exploring how to generate sound arguments from identified fallacies and their context, and how to counter invalid arguments.
  • Diachronic Analysis: While the paper presents some diachronic analysis, further investigation into the evolution of debating strategies over time is suggested.

Summary updated

BibTeX
@inproceedings{goffredo2022fallacious,
  author = {Pierpaolo Goffredo and Shohreh Haddadan and Vorakit Vorakitphan and Elena Cabrio and Serena Villata},
  title = {{Fallacious Argument Classification in Political Debates}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/575},
}
Figure