Towards Effective Rebuttal: Listening Comprehension using Corpus-Wide Claim Mining

Tamar Lavee, Matan Orbach, Lili Kotlerman, Yoav Kantor, Shai Gretz, Lena Dankin, Shachar Mirkin, Michal Jacovi, Yonatan Bilu, Ranit Aharonov, Noam Slonim

TL;DR

To effectively rebut an opponent in a live debate, an AI must first understand which arguments were actually made. This paper proposes a "corpus-wide claim mining" approach: automatically extracting millions of claims from a massive news corpus and matching them to spontaneous spoken speeches. The authors release a new dataset of 400 speeches and demonstrate that while most human arguments are made implicitly, they can be successfully identified using semantic similarity baselines.

What Problem Does This Paper Address?

In the context of IBM’s Project Debater, the goal is to create a system that can engage in a full, live debate with human experts. A critical bottleneck is "listening comprehension"—specifically, identifying the specific arguments an opponent has voiced so the system can generate a relevant rebuttal.

Traditional approaches rely on manually curated databases like iDebate, which suffer from low topic coverage and a limited number of arguments per side. The authors tackle two main challenges:

  1. Coverage: Can claims mined from a general-purpose news corpus (billions of sentences) actually represent the nuanced arguments made by human debaters in real-time?
  2. Matching: How do we automatically detect these claims in spoken speech, which is often messy, filled with disfluencies (like "um" or repetitions), and prone to errors from Automatic Speech Recognition (ASR) systems?

Methods

The researchers developed a pipeline that moves from a massive text corpus to a targeted rebuttal, as illustrated in the system architecture:

Figure 1: The suggested architecture for mined–claims based rebuttal generation. System inputs are depicted with a gray background. The focus of this work is marked on the right: detecting mentioned claims in opponents speeches. Preceding existing components are shortly described in Section 3. The entire pipeline starts from billions of sentences, and its final goal is producing few high quality rebuttals opposing the opponent speech.

The process involves several key stages:

  • Claim Mining: They start with a corpus of ~10 billion sentences from news articles. For any given topic (e.g., "We should introduce compulsory voting"), the system retrieves relevant sentences, ranks them using a neural model, detects the exact boundaries of the claim, and determines its stance (pro or con).
  • Claim Matching: This is the core focus of the paper. Once they have a set of "Opposing Claims," they try to match them against the "Opponent Speech."
  • Baselines: They tested three matching approaches:
  • HM (Harmonic Mean): An unsupervised baseline that averages three semantic similarity measures: Concept Coverage (Wikipedia concepts), Parse Pairs (structural similarity of parse trees), and Explicit Semantic Analysis (ESA).
  • LR (Logistic Regression): A supervised model using 23 features, including word2vec embeddings and part-of-speech similarities.
  • NN (Siamese Network): A deep learning approach using BiLSTMs with an attention layer to score pairs of sentences and claims.

Datasets

The authors introduced a significant new benchmark for the community:

  • 400 Speeches: Recorded by 9 expert debaters covering 200 controversial topics.
  • Transcripts: Both manual "reference" transcripts and automatic ASR transcripts (with a 7.07% word error rate).
  • Annotated Pairs: 4,882 speech-claim pairs. Each pair was labeled by seven crowd-sourced annotators as Explicitly mentioned, Implicitly mentioned, or Not mentioned.
  • Comparison: They noted that their mined claims provided candidates for 93.5% of motions, whereas the existing iDebate-based dataset only covered 39%.

Key Insights and Ideas

  • The "Implicit" Challenge: A major finding was that only 5.6% of mentions were explicit. Human debaters rarely state a claim word-for-word as it appears in a news article; instead, they allude to it or build up to it over several sentences. This makes the matching task significantly harder than simple keyword search.
  • Corpus Mining Works: Despite the differences between written news and spoken debate, the news-mined claims had excellent coverage. In 87.3% of the speeches, at least one mined claim was identified by annotators.
  • Simplicity vs. Complexity: Surprisingly, the unsupervised HM baseline outperformed the more complex supervised models (LR and NN). The authors suggest this indicates the difficulty of the task and the potential need for more sophisticated supervised methods or larger training sets to capture the nuances of implicit mentions.

Key Results

  • Coverage: On average, the system mined 12.2 claims per speech, with about a third (4.2) being confirmed as mentioned by annotators.
  • Performance: The HM configuration achieved an Area Under the ROC Curve (AUC) higher than 0.62, whereas the supervised LR and NN models failed to exceed 0.57 on the training set.
  • Precision-Recall: As shown in Figure 2, the $K=1$ configuration (matching the claim to the single most similar sentence in a speech) showed a slight advantage for applications where precision is more important than recall.
Figure 2: Precision-Recall curves for the top-3 claim matching configurations (all using HM) on test.

Limitations and Future Work

The authors acknowledge that the high lexical variability of implicit claims remains a hurdle. Because a claim might be spread across multiple non-consecutive sentences, current sentence-level matching models are limited.

Future work will likely focus on:

  1. Improving supervised matching models to better handle "alluded to" arguments.
  2. Developing methods to handle ASR errors more robustly.
  3. Moving from identifying the claim to actually generating the rebuttal, which is the final step in the Project Debater pipeline.

Summary updated

Abstract

Engaging in a live debate requires, among other things, the ability to effectively rebut arguments claimed by your opponent. In particular, this requires identifying these arguments. Here, we suggest doing so by automatically mining claims from a corpus of news articles containing billions of sentences, and searching for them in a given speech. This raises the question of whether such claims indeed correspond to those made in spoken speeches. To this end, we collected a large dataset of $400$ speeches in English discussing $200$ controversial topics, mined claims for each topic, and asked annotators to identify the mined claims mentioned in each speech. Results show that in the vast majority of speeches debaters indeed make use of such claims. In addition, we present several baselines for the automatic detection of mined claims in speeches, forming the basis for future work. All collected data is freely available for research.
BibTeX
@inproceedings{lavee2019towards,
  author = {Tamar Lavee and Matan Orbach and Lili Kotlerman and Yoav Kantor and Shai Gretz and Lena Dankin and Shachar Mirkin and Michal Jacovi and Yonatan Bilu and Ranit Aharonov and Noam Slonim},
  title = {{Towards Effective Rebuttal: Listening Comprehension using Corpus-Wide Claim Mining}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}

Loading related content...

Figure