Never Retreat, Never Retract: Argumentation Analysis for Political Speeches

Stefano Menini, Elena Cabrio, Sara Tonelli, Serena VillataAAAI 2018 · DOI

Abstract

Political discourse often unfolds across disconnected speeches rather than direct debates. This paper presents an argumentation mining pipeline designed to reconstruct these "hidden" dialogues by automatically identifying support and attack relations between claims in monological political speeches. Using the 1960 Nixon-Kennedy campaign as a case study, the researchers demonstrate how machine learning can map the complex argumentative landscape of historical campaigns.

Introduction

In the digital age, historians and political scientists are increasingly overwhelmed by the sheer volume of archival text. Analyzing these documents qualitatively to identify points of agreement or contention between candidates is a labor-intensive process. While existing argument mining tools excel at analyzing direct interactions—such as live debates or social media threads—they often struggle with political speeches. These speeches are typically monological, where the "dialogue" between opponents is indirect, occurring across different times and locations.

To bridge this gap, this research introduces a pipeline that detects these hidden relations across disconnected documents. By automating the identification of argumentative structures, the system provides a comprehensive map of a campaign's discourse, offering a powerful new lens for historical and social science research.

The Argumentation Mining Pipeline

The proposed system utilizes a two-stage machine learning pipeline based on Support Vector Machines (SVM) to classify the relationships between argument snippets.

1. Argument Extraction

The process begins by extracting three-sentence snippets—comprising a central claim and its immediate context—from a large corpus of campaign documents. This extraction is keyword-driven, focusing on five pivotal topics: healthcare, Cuba, disarmament, minimum wage, and unemployment.

2. Relation Prediction

The core of the pipeline involves a two-step classification process:

  • Step 1 (Related vs. Unrelated): A binary classifier filters pairs to identify those that genuinely address one another. This step relies on lexical overlap (shared nouns and verbs) and the positional context of keywords.
  • Step 2 (Support vs. Attack): For pairs identified as related, a second classifier determines whether the relationship is one of support or attack. This model incorporates advanced features such as sentiment analysis, negation detection, keyword embeddings (word2vec), and logical entailment.

The resulting data is formatted using the Argument Interchange Format (AIF) and visualized through OVA+, creating an interactive and searchable graph of the campaign's arguments.

Figure 1: An interactive argument graph generated by OVA+, visualizing the support and attack relations between Nixon and Kennedy's campaign statements.

Dataset and Topic Analysis

The researchers developed a manually annotated benchmark dataset from 881 documents of the 1960 US Presidential campaign. The corpus, totaling 1.6 million tokens, yielded 1,462 annotated pairs of arguments. The dataset is balanced across "Attack" (378), "Support" (353), and "No Relation" (731) categories.

Analysis of the topics revealed that the system's performance varies based on the nature of the subject matter. Concrete issues, such as the minimum wage, allowed for higher accuracy (0.94 in Step 2) due to the specific and identifiable language used in those debates. In contrast, abstract topics like "disarmament" often involve "fuzzy" vocabulary shared by both sides, making relation detection more challenging.

Key Insights and Results

The study highlights several important findings regarding political argumentation:

  • Reconstructing Indirect Dialogue: The research proves that a coherent "debate" can be mapped even when participants are not physically present together. By chronologically pairing statements, the system captures the evolution of political arguments over time.
  • Linguistic Consistency: The model demonstrated higher accuracy (0.86) when predicting relations between statements made by the same speaker compared to different speakers (0.79). This suggests that candidates maintain a high degree of linguistic consistency when building upon their own previous arguments.

Quantitative Performance

In a real-world application where the system must first determine if a pair is related, it achieved an overall accuracy of 0.72. However, when evaluated on a "gold standard" set of pairs known to be related, the accuracy for the Support/Attack classification rose to 0.82, significantly outperforming the random baseline of 0.51.

Figure 5: Performance metrics for the Support/Attack classifier on the gold standard dataset, demonstrating robust precision and recall across both categories.

Limitations and Future Work

Despite its success, the pipeline has areas for improvement. Currently, the system treats three-sentence snippets as monolithic arguments. Future iterations aim for finer-grained detection to distinguish between specific claims and the evidence supporting them. Additionally, the model still faces challenges with complex rhetorical devices like double negatives and rhetorical questions.

The ultimate goal is to conduct qualitative user evaluations with historians to ensure these automated visualizations provide meaningful utility for scholarly research.

Summary updated

BibTeX
@inproceedings{menini2018never,
  author = {Stefano Menini and Elena Cabrio and Sara Tonelli and Serena Villata},
  title = {{Never Retreat, Never Retract: Argumentation Analysis for Political Speeches}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11920},
}
Figure