Automatic Claim Negation: Why, How and When

Yonatan Bilu, Daniel Hershcovich, Noam Slonim

Abstract

Researchers at IBM Haifa Research Lab have developed a two-stage framework to automatically generate counter-claims by negating existing ones. By combining a rule-based linguistic engine with a machine learning classifier, the system can not only flip a claim's meaning but also predict whether the resulting sentence is "usable" and plausible in a real-world debate. The study identifies a hierarchy of negation quality, demonstrating that while grammatical inversion is relatively straightforward, achieving debate-ready usability remains a significant challenge.

Introduction

In the field of Argumentation Mining, systems typically focus on identifying claims and evidence already present in a text. However, to build a truly robust debate-support system, it is necessary to explore the "other side." The authors identify a gap in current technology: the automatic generation of claim negations.

A generated negation must satisfy four increasingly difficult criteria to be effective:

  1. Grammar: Is it syntactically correct?
  2. Clarity: Is it easy to understand?
  3. Opposition: Does it actually state the opposite of the original?
  4. Usability: Would a human actually say this in a debate?

Methods

The authors propose a two-stage "How and When" approach to address these criteria.

The "How": Rule-Based Negation

The first stage uses a set of linguistic rules to transform an input claim. Utilizing the Stanford CoreNLP pipeline, the system identifies parts of speech and applies logic such as:

  • Modal/Auxiliary Verbs: Changing "is" to "is not" or "can't" to "can."
  • Prefixes: Using WordNet to identify antonyms or remove negation prefixes (e.g., changing "unworkable" to "workable").
  • Lemmatization: Handling verb conjugations when inserting "does not" (e.g., "The law requires" becomes "The law does not require").

The "When": Statistical Usability Filter

Because rule-based engines often produce technically correct but awkward sentences, the authors trained a Logistic Regression classifier to predict usability. The classifier utilizes 19 features, including structural features (word counts, POS tags), sentiment analysis, and N-gram frequency. The latter acts as a "sanity check" by comparing the frequency of the generated phrase in Google n-grams against the original; if the negation is non-existent in real-world usage, it is flagged as a poor candidate.

Key Results and Insights

The study utilized a dataset of 1,240 claims from Wikipedia. Evaluation by five annotators revealed that the criteria form a natural hierarchy. While the rule-based system was excellent at maintaining grammar, usability proved to be the primary bottleneck.

Performance of the rule-based engine across the four evaluation criteria.

As shown in the table above, the rule-based engine achieved 96% success in Grammar but only 50% in Usability. This confirms that "meaningful contradiction" is a much higher bar than simple linguistic inversion.

To improve these results, the Logistic Regression filter was applied. The classifier significantly improved the quality of the output by prioritizing high-confidence results.

Mean precision of the usability classifier relative to the rank of the generated claims.

When the classifier selected its top 5 most confident negations, 86% were deemed usable by humans. Even within the top 20 results, precision remained high at 78%, as illustrated in the precision-rank graph.

Discussion and Future Work

The authors noted that for approximately 75% of claims that can be negated, simple rules (adding/removing "not") are sufficient, provided the system can correctly identify when to apply them. However, limitations remain. The system occasionally struggles with the nuance of modal verbs (e.g., "must" vs "must not") and currently only targets a single scope of negation within a sentence.

Future work will focus on handling more complex claims where negating different parts of the sentence might produce a more natural counter-argument. Additionally, the authors aim to link these generated claims to actual supporting evidence, as a negated claim is only useful in a debate if it can be backed by data.

Summary updated

Abstract

The main goal of argumentation mining is to analyze argumentative structures within an argument-rich document, and reason about their composition. Recently, there is also interest in the task of simply detecting claims (sometimes called conclusion) in general documents. In this work we ask how this set of detected claims can be augmented further, by adding to it the negation of each detected claim. This presents two NLP problems: how to automatically negate a claim, and when such a negated claim can plausibly be used. We present first steps into solving both these problems, using a rule-based approach for the former and a statistical one towards the latter.
BibTeX
@inproceedings{bilu2015automatic,
  author = {Yonatan Bilu and Daniel Hershcovich and Noam Slonim},
  title = {{Automatic Claim Negation: Why, How and When}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/w15-0511},
}
Figure