QED: A fact verification system for the FEVER shared task

Jackson Luken, Nanjiang Jiang, Marie-Catherine de MarneffeEMNLP 2018

TL;DR

This paper introduces QED, a fact verification system designed for the FEVER shared task. QED tackles the challenge of automatically verifying or refuting factual claims using evidence from Wikipedia, achieving a significant improvement over the baseline and demonstrating strong evidence retrieval capabilities.

What Problem Does This Paper Address?

The paper addresses the problem of automated fact verification, specifically within the context of the FEVER (Fact Extraction and VERification) shared task. The goal is to build a system that can take a factual claim and determine if it is supported, refuted, or if there's not enough information (NEI) to make a judgment, all while providing valid evidence from a large corpus like Wikipedia. This is a crucial task for combating misinformation and ensuring the reliability of information in the digital age. The challenge lies in accurately identifying relevant evidence from a vast amount of text and then correctly inferring the relationship between that evidence and the claim.

Methods

QED employs a three-module pipeline:

  1. Document Finder: This module identifies potentially relevant Wikipedia articles by extracting key phrases from the claim and matching them against Wikipedia article titles. It uses a multi-map approach (case-sensitive, lowercase, and parenthesis-stripped titles) to improve recall, even at the cost of some precision, by including backup documents.
  2. Sentence Finder: This module compares sentences within the retrieved documents against the claim to find potential evidence. It uses a combination of matching the claim's root verb and identifying sentences that contain a sufficient number of nouns and named entities from the claim. Special handling is included for cases with missing entities or when dates are involved in the claim.
  3. Inference Module: This module uses a decomposable attention model (Parikh et al., 2016) to classify the relationship between the claim and the retrieved evidence sentences as "Supports," "Refutes," or "NEI." Preprocessing involves deterministic coreference resolution for pronouns and replacing them with the document's title. Named entities and their types are extracted and fed into embedding layers, combined with GloVe word embeddings, as input to the network.

Datasets

  • FEVER dataset: Used for both training and evaluation. The development set consists of 19,998 claims (6,666 in each class: Supports, Refutes, NEI).
  • Wikipedia: Serves as the source corpus for retrieving evidence.
  • SNLI corpus: Mentioned as a benchmark for the inference model.
  • OntoNotes: Used for training the named entity recognizer.
  • CommonCrawl: Used for pre-training GloVe word embeddings.

Key Insights and Ideas

  • Prioritizing Recall in Evidence Retrieval: The system intentionally sacrifices precision in the Document Finder stage to maximize recall, ensuring that as many potentially relevant documents as possible are considered. This is a strategic choice, as irrelevant documents will be filtered out in later stages.
  • Leveraging Document Context for Sentence Finding: The Sentence Finder intelligently discards nouns and named entities from the claim that are already present in the document's title. This is based on the assumption that sentences within a document naturally relate to its topic, reducing redundancy and focusing on novel information.
  • Handling Ambiguity and Coreference: The system's preprocessing for the inference module, particularly the deterministic coreference resolution, is crucial for handling claims that might refer to entities using pronouns or descriptions rather than explicit names.
  • Annotation Artifacts in NLI: The paper touches upon the idea that NLI datasets can contain "annotation artifacts," where certain words (like "never") can be strong indicators of a particular class ("Refutes") even if the overall context is more nuanced. This highlights a challenge in building robust NLI models.

Key Results

  • FEVER Score: The system achieves a FEVER score of 43.9% on the development set and 43.4% on the test set, outperforming the baseline system (31.3% on development, 27.5% on test).
  • Evidence Retrieval F1: QED demonstrates strong evidence retrieval, with an F1 score of 62.5% on the development set, significantly exceeding the baseline's 17.2%. On the test set, it ranked 2nd with an F1 of 58.5%.
  • Label Accuracy: Label accuracy on the development set was 44.7% for the system, compared to 51.4% for the baseline. This indicates that while evidence retrieval improved, the final classification accuracy was not as dramatically enhanced.
  • Misclassification Patterns: Table 2 reveals that "Refutes" and "NEI" claims are often mislabeled as "Supports," and "Supports" claims are frequently mislabeled as "NEI."

Limitations and Future Work

  • Evidence Retrieval Precision: While recall is high, the precision of the Document Finder is relatively low (7.5% when backup documents are included), largely due to disambiguation pages.
  • Sentence Finder Recall: The Sentence Finder fails to return evidence for a notable percentage of verifiable claims (24.5%), often due to misclassifications of named entities or part-of-speech tags by the NLP pipeline, or unseen contextual connections.
  • Lexical Semantics and World Knowledge: The inference model struggles with claims requiring fine-grained lexical semantics and world knowledge, as exemplified by the "Gin is a drink" claim being misclassified.
  • Annotation Artifacts: The paper acknowledges the potential for annotation artifacts in NLI datasets, suggesting that negation words can be overly strong indicators for the "Refutes" class.
  • Future Work: The authors suggest exploring the NEARESTP sampling method for inference and investigating patterns in correctly retrieved evidence that is not present in the gold standard. They also note that working on lemma could help with number mismatches in document titles.

Summary updated

BibTeX
@inproceedings{luken2018qed,
  author = {Jackson Luken and Nanjiang Jiang and Marie-Catherine de Marneffe},
  title = {{QED: A fact verification system for the FEVER shared task}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure