End-to-End Argumentation Knowledge Graph Construction

Khalid Al Khatib, Yufang Hou, Henning Wachsmuth, Charles Jochim, Francesca Bonin, Benno SteinAAAI 2020 · OpenReview

Abstract

This paper studies the end-to-end construction of an argumentation knowledge graph that is intended to support argument synthesis, argumentative question answering, or fake news detection, among others. The study is motivated by the proven effectiveness of knowledge graphs for interpretable and controllable text generation and exploratory search. Original in our work is that we propose a model of the knowledge encapsulated in arguments. Based on this model, we build a new corpus that comprises about 16k manual annotations of 4740 claims with instances of the model's elements, and we develop an end-to-end framework that automatically identifies all modeled types of instances. The results of experiments show the potential of the framework for building a web-based argumentation graph that is of high quality and large scale.

Introduction

While traditional Knowledge Graphs (KGs) like ConceptNet or Freebase are excellent at storing factual triples (e.g., "Paris is the capital of France"), they often lack the structured representation of human reasoning and argumentation. To build AI systems capable of synthesizing persuasive essays or detecting fallacious reasoning, we require a structured way to represent how ideas interact. For instance, knowing that "cycling" is an activity is insufficient; an argumentation system must understand that "cycling" has a positive effect on "health" and a negative effect on "pollution."

The Argumentation Model

The researchers propose a novel model for argumentation KGs defined by three primary components:

  • Nodes (Concept Instances): Representing entities, events, or abstract ideas.
  • Edges (Effect Relations): Capturing whether one concept promotes (positive) or suppresses (negative) another.
  • Attributes: Including "concept consequences" (classifying a concept as generally "good" or "bad") and "groundings" (linking concepts to Wikipedia entries to resolve synonyms).
An example of an argumentation knowledge graph showing relations between concepts like cycling, health, and nuclear energy.

Methodology

To realize this model, the authors followed a three-step process:

  1. Manual Acquisition: Using crowdsourcing via Amazon Mechanical Turk, the team sampled 4,740 claims from Debatepedia to create a corpus of 16,429 manual annotations. This dataset identifies concepts and the nature of their relationships.
  2. Automatic Framework: A supervised learning pipeline was developed using Support Vector Machines (SVM). The system leverages lexical (n-grams), syntactic (POS tags), sentiment, and semantic features (using lexicons like EffectWordNet).
  3. Concept Identification: The framework utilizes Open Information Extraction (OpenIE) and Semantic Role Labeling (SRL) to extract the "actor" and "object" within argumentative sentences.

Key Results and Generalization

The framework demonstrated strong performance in detecting and classifying relations. It achieved a macro $F_1$-score of 0.79 for relation detection and 0.77 for type classification (positive vs. negative).

Crucially, the system showed an ability to generalize to unseen data. When applied to topics from Wikipedia and the Annotated English Gigaword, the framework maintained an average precision of 0.69 across diverse subjects such as Global Warming and Social Security.

Precision of the framework across various unseen topics, demonstrating its generalization capabilities.

Insights and Future Work

A key finding was that simple lexical features (n-grams) often outperformed complex syntactic features, suggesting that specific "cue words" (e.g., "promotes," "undermines") are highly reliable indicators of argumentative structure. Furthermore, grounding concepts in Wikipedia allows the graph to infer "hidden" relationships across different documents.

However, limitations remain in "consequence classification" (determining if a concept is inherently good or bad), which achieved a lower $F_1$ of 0.49 due to the subjective nature of value judgments. Future work aims to scale this pipeline to construct a massive, web-based argumentation graph from general web text.

Summary updated

Abstract

This paper studies the end-to-end construction of an argumentation knowledge graph that is intended to support argument synthesis, argumentative question answering, or fake news detection, among others. The study is motivated by the proven effectiveness of knowledge graphs for interpretable and controllable text generation and exploratory search. Original in our work is that we propose a model of the knowledge encapsulated in arguments. Based on this model, we build a new corpus that comprises about 16k manual annotations of 4740 claims with instances of the model's elements, and we develop an end-to-end framework that automatically identifies all modeled types of instances. The results of experiments show the potential of the framework for building a web-based argumentation graph that is of high quality and large scale.
BibTeX
@inproceedings{khatib2020end,
  author = {Khalid Al Khatib and Yufang Hou and Henning Wachsmuth and Charles Jochim and Francesca Bonin and Benno Stein},
  title = {{End-to-End Argumentation Knowledge Graph Construction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=vkhs9HltWXo},
}
Figure