TL;DR
To effectively rebut an opponent in a live debate, an AI must first understand which arguments were actually made. This paper proposes a "corpus-wide claim mining" approach: automatically extracting millions of claims from a massive news corpus and matching them to spontaneous spoken speeches. The authors release a new dataset of 400 speeches and demonstrate that while most human arguments are made implicitly, they can be successfully identified using semantic similarity baselines.
What Problem Does This Paper Address?
In the context of IBM’s Project Debater, the goal is to create a system that can engage in a full, live debate with human experts. A critical bottleneck is "listening comprehension"—specifically, identifying the specific arguments an opponent has voiced so the system can generate a relevant rebuttal.
Traditional approaches rely on manually curated databases like iDebate, which suffer from low topic coverage and a limited number of arguments per side. The authors tackle two main challenges:
- Coverage: Can claims mined from a general-purpose news corpus (billions of sentences) actually represent the nuanced arguments made by human debaters in real-time?
- Matching: How do we automatically detect these claims in spoken speech, which is often messy, filled with disfluencies (like "um" or repetitions), and prone to errors from Automatic Speech Recognition (ASR) systems?
Methods
The researchers developed a pipeline that moves from a massive text corpus to a targeted rebuttal, as illustrated in the system architecture:
The process involves several key stages:
- Claim Mining: They start with a corpus of ~10 billion sentences from news articles. For any given topic (e.g., "We should introduce compulsory voting"), the system retrieves relevant sentences, ranks them using a neural model, detects the exact boundaries of the claim, and determines its stance (pro or con).
- Claim Matching: This is the core focus of the paper. Once they have a set of "Opposing Claims," they try to match them against the "Opponent Speech."
- Baselines: They tested three matching approaches:
- HM (Harmonic Mean): An unsupervised baseline that averages three semantic similarity measures: Concept Coverage (Wikipedia concepts), Parse Pairs (structural similarity of parse trees), and Explicit Semantic Analysis (ESA).
- LR (Logistic Regression): A supervised model using 23 features, including word2vec embeddings and part-of-speech similarities.
- NN (Siamese Network): A deep learning approach using BiLSTMs with an attention layer to score pairs of sentences and claims.
Datasets
The authors introduced a significant new benchmark for the community:
- 400 Speeches: Recorded by 9 expert debaters covering 200 controversial topics.
- Transcripts: Both manual "reference" transcripts and automatic ASR transcripts (with a 7.07% word error rate).
- Annotated Pairs: 4,882 speech-claim pairs. Each pair was labeled by seven crowd-sourced annotators as Explicitly mentioned, Implicitly mentioned, or Not mentioned.
- Comparison: They noted that their mined claims provided candidates for 93.5% of motions, whereas the existing iDebate-based dataset only covered 39%.
Key Insights and Ideas
- The "Implicit" Challenge: A major finding was that only 5.6% of mentions were explicit. Human debaters rarely state a claim word-for-word as it appears in a news article; instead, they allude to it or build up to it over several sentences. This makes the matching task significantly harder than simple keyword search.
- Corpus Mining Works: Despite the differences between written news and spoken debate, the news-mined claims had excellent coverage. In 87.3% of the speeches, at least one mined claim was identified by annotators.
- Simplicity vs. Complexity: Surprisingly, the unsupervised HM baseline outperformed the more complex supervised models (LR and NN). The authors suggest this indicates the difficulty of the task and the potential need for more sophisticated supervised methods or larger training sets to capture the nuances of implicit mentions.
Key Results
- Coverage: On average, the system mined 12.2 claims per speech, with about a third (4.2) being confirmed as mentioned by annotators.
- Performance: The HM configuration achieved an Area Under the ROC Curve (AUC) higher than 0.62, whereas the supervised LR and NN models failed to exceed 0.57 on the training set.
- Precision-Recall: As shown in Figure 2, the $K=1$ configuration (matching the claim to the single most similar sentence in a speech) showed a slight advantage for applications where precision is more important than recall.
Limitations and Future Work
The authors acknowledge that the high lexical variability of implicit claims remains a hurdle. Because a claim might be spread across multiple non-consecutive sentences, current sentence-level matching models are limited.
Future work will likely focus on:
- Improving supervised matching models to better handle "alluded to" arguments.
- Developing methods to handle ASR errors more robustly.
- Moving from identifying the claim to actually generating the rebuttal, which is the final step in the Project Debater pipeline.
Summary updated