Automated Fact-Checking of Claims in Argumentative Parliamentary Debates

Nona Naderi, Graeme HirstEMNLP 2018

TL;DR

This paper introduces an automated approach to classify claims in Canadian parliamentary debates into four categories: true, false, dodge, and stretch. By leveraging linguistic features and incorporating predictions from a model trained on US fact-checking data, they achieve promising results, particularly in identifying "dodge" statements.

What Problem Does This Paper Address?

Governments and politicians often employ a mix of true, false, and exaggerated statements to influence public perception, gain credibility, or discredit opponents. Distinguishing between these types of claims is crucial for understanding political discourse and holding public figures accountable. While automated fact-checking has been explored for general claims, applying it to the specific context of argumentative parliamentary debates, with their unique linguistic nuances and the presence of "dodge" statements (where a direct answer is avoided), presents a significant challenge. This paper aims to bridge this gap by developing and evaluating an automated system for classifying such claims.

Methods

The authors approach this problem as a multi-class classification task. They experiment with various linguistic features previously found effective in identifying deceptive texts, including:

  • Bag-of-Words (BOW) with TF-IDF: Standard text representation.
  • Part-of-Speech (POS) tags: Analyzing the grammatical structure.
  • Number of words: Simple length-based features.
  • Named Entity Recognition (NER): Identifying organizations and locations.
  • Numeric counts: The presence of numbers in statements.
  • LIWC (Linguistic Inquiry and Word Count): Analyzing psychological and linguistic dimensions of text.
  • Intensifying words: Lexicons of superlatives, comparatives, etc.

A key innovation is the incorporation of truthfulness predictions from a Gated Recurrent Unit (GRU) classifier trained on the PolitiFact dataset. These probabilities are used as additional features for an SVM classifier. This allows them to leverage knowledge from a large US fact-checking corpus to aid in classifying Canadian parliamentary statements. They also explore pairwise classification to better understand the impact of features on specific label distinctions.

Datasets

  • Canadian Parliament Oral Question Period (Toronto Star dataset): This dataset, collected by the Toronto Star newspaper, consists of annotated questions and answers from the Canadian Parliament. Statements are labeled as true, false, stretch, half true, or dodge. The paper notes this dataset is skewed, with a majority of statements labeled as true.
  • PolitiFact dataset: A publicly available dataset from the US fact-checking website PolitiFact, featuring a 6-point scale of truthfulness. This dataset is used to train the GRU model that provides additional features for the Canadian dataset.

Key Insights and Ideas

  • "Dodge" statements are surprisingly detectable: The most significant finding is the high accuracy in identifying "dodge" statements, suggesting that evasive language has distinct linguistic markers.
  • Leveraging external data is beneficial: Incorporating predictions from a model trained on a large US fact-checking dataset (PolitiFact) improved the classification performance on the Canadian parliamentary data, even though the datasets and annotation schemes differ.
  • Not all linguistic features are equally useful: While many features were tested, only a subset, particularly named entities and superlatives, showed significant impact on classification accuracy.
  • Stretch statements are challenging: The paper highlights that "stretch" statements were the most difficult to classify in binary settings, potentially due to ambiguity in annotation or inherent linguistic similarity to other categories.

Key Results

  • The best multi-class classification model, using a combination of unigrams, POS tags, number counts, superlatives, and PolitiFact predictions, achieved improved results over baselines.
  • In pairwise classification, "dodge" statements were classified with an F₁ measure as high as 82.57%.
  • Named entities proved to be the most effective features for distinguishing between "false" and "true" or "dodge" statements.
  • The GRU predictions from PolitiFact data alone were not sufficient to distinguish "false" from "true" and "stretch" statements on the Canadian data.

Limitations and Future Work

  • Annotation ambiguity: The authors acknowledge that "stretch" statements were the most difficult to classify, possibly due to inconsistencies in the annotation process and the inherent difficulty in separating "stretch" from "false."
  • Reliance on linguistic features: The paper suggests that relying solely on linguistic features might not be sufficient for determining truthfulness.
  • Future work: The authors plan to incorporate journalists' justifications for their annotations. This qualitative information could provide deeper insights into the reasoning behind truthfulness judgments and potentially improve the model's ability to detect falsehoods. They also aim to explore more sophisticated methods for integrating external knowledge.

Summary updated

BibTeX
@inproceedings{naderi2018automated,
  author = {Nona Naderi and Graeme Hirst},
  title = {{Automated Fact-Checking of Claims in Argumentative Parliamentary Debates}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure