Abstract
Political discourse often unfolds across disconnected speeches rather than direct debates. This paper presents an argumentation mining pipeline designed to reconstruct these "hidden" dialogues by automatically identifying support and attack relations between claims in monological political speeches. Using the 1960 Nixon-Kennedy campaign as a case study, the researchers demonstrate how machine learning can map the complex argumentative landscape of historical campaigns.
Introduction
In the digital age, historians and political scientists are increasingly overwhelmed by the sheer volume of archival text. Analyzing these documents qualitatively to identify points of agreement or contention between candidates is a labor-intensive process. While existing argument mining tools excel at analyzing direct interactions—such as live debates or social media threads—they often struggle with political speeches. These speeches are typically monological, where the "dialogue" between opponents is indirect, occurring across different times and locations.
To bridge this gap, this research introduces a pipeline that detects these hidden relations across disconnected documents. By automating the identification of argumentative structures, the system provides a comprehensive map of a campaign's discourse, offering a powerful new lens for historical and social science research.
The Argumentation Mining Pipeline
The proposed system utilizes a two-stage machine learning pipeline based on Support Vector Machines (SVM) to classify the relationships between argument snippets.
1. Argument Extraction
The process begins by extracting three-sentence snippets—comprising a central claim and its immediate context—from a large corpus of campaign documents. This extraction is keyword-driven, focusing on five pivotal topics: healthcare, Cuba, disarmament, minimum wage, and unemployment.
2. Relation Prediction
The core of the pipeline involves a two-step classification process:
- Step 1 (Related vs. Unrelated): A binary classifier filters pairs to identify those that genuinely address one another. This step relies on lexical overlap (shared nouns and verbs) and the positional context of keywords.
- Step 2 (Support vs. Attack): For pairs identified as related, a second classifier determines whether the relationship is one of support or attack. This model incorporates advanced features such as sentiment analysis, negation detection, keyword embeddings (word2vec), and logical entailment.
The resulting data is formatted using the Argument Interchange Format (AIF) and visualized through OVA+, creating an interactive and searchable graph of the campaign's arguments.
Dataset and Topic Analysis
The researchers developed a manually annotated benchmark dataset from 881 documents of the 1960 US Presidential campaign. The corpus, totaling 1.6 million tokens, yielded 1,462 annotated pairs of arguments. The dataset is balanced across "Attack" (378), "Support" (353), and "No Relation" (731) categories.
Analysis of the topics revealed that the system's performance varies based on the nature of the subject matter. Concrete issues, such as the minimum wage, allowed for higher accuracy (0.94 in Step 2) due to the specific and identifiable language used in those debates. In contrast, abstract topics like "disarmament" often involve "fuzzy" vocabulary shared by both sides, making relation detection more challenging.
Key Insights and Results
The study highlights several important findings regarding political argumentation:
- Reconstructing Indirect Dialogue: The research proves that a coherent "debate" can be mapped even when participants are not physically present together. By chronologically pairing statements, the system captures the evolution of political arguments over time.
- Linguistic Consistency: The model demonstrated higher accuracy (0.86) when predicting relations between statements made by the same speaker compared to different speakers (0.79). This suggests that candidates maintain a high degree of linguistic consistency when building upon their own previous arguments.
Quantitative Performance
In a real-world application where the system must first determine if a pair is related, it achieved an overall accuracy of 0.72. However, when evaluated on a "gold standard" set of pairs known to be related, the accuracy for the Support/Attack classification rose to 0.82, significantly outperforming the random baseline of 0.51.
Limitations and Future Work
Despite its success, the pipeline has areas for improvement. Currently, the system treats three-sentence snippets as monolithic arguments. Future iterations aim for finer-grained detection to distinguish between specific claims and the evidence supporting them. Additionally, the model still faces challenges with complex rhetorical devices like double negatives and rhetorical questions.
The ultimate goal is to conduct qualitative user evaluations with historians to ensure these automated visualizations provide meaningful utility for scholarly research.
Summary updated