DeFactoNLP: Fact Verification using Entity Recognition, TFIDF Vector Comparison and Decomposable Attention

Aniketh Janardhan Reddy, Gil Rocha, Diego EstevesEMNLP 2018

TL;DR

This paper introduces DeFactoNLP, a system for automated fact verification that assesses the truthfulness of a claim and retrieves supporting evidence from Wikipedia. It combines entity recognition, TF-IDF similarity, and a decomposable attention-based textual entailment model to achieve competitive results on the FEVER benchmark.

What Problem Does This Paper Address?

The proliferation of fake news on social media necessitates automated fact-checking systems. These systems must not only determine the veracity of a claim but also identify and present the evidence supporting that determination. This is a complex task requiring the retrieval of relevant information, the analysis of its relationship to the claim, and a final classification. The FEVER (Fact Extraction and Verification) dataset and shared task were created to address this challenge by providing a large-scale benchmark for training and evaluating such systems.

Methods

DeFactoNLP employs a four-step architecture:

  1. Relevant Document Retrieval: This step identifies Wikipedia documents that are likely to contain evidence. Two methods are used:
  2. Named Entity Recognition (NER)-based: It identifies named entities in the claim and retrieves Wikipedia documents whose titles are most similar (using Levenshtein distance) to these entities.
  3. TF-IDF Vector Comparison-based: It retrieves documents whose TF-IDF vectors are most similar to the claim's TF-IDF vector. The system then selects the top five most similar documents.
  4. Relevant Sentence Retrieval: From the retrieved documents, the system identifies the five most similar sentences to the claim based on TF-IDF vector similarity.
  5. Textual Entailment Recognition: A fine-tuned decomposable attention model (based on Peters et al., 2018) is used to determine the relationship between each candidate sentence and the claim. It outputs probabilities for the sentence supporting ("Entailment"), contradicting ("Contradiction"), or being uninformative ("Neutral") with respect to the claim. The model was fine-tuned on a custom FEVER SNLI-style dataset to improve generalization.
  6. Final Classification: Twelve features are derived from the entailment probabilities. These features, which capture the number and strength of supporting/refuting sentences, are fed into a Random Forest classifier to assign a final label ("SUPPORTS," "REFUTES," or "NOT ENOUGH INFO") to the claim. The system also returns supporting sentences as evidence.

Datasets

  • FEVER Dataset: Used for the shared task and as the basis for creating a custom training dataset.
  • SNLI Corpus: Used for initial training of the Textual Entailment Recognition (RTE) model.
  • FEVER SNLI-style Dataset: A newly synthesized dataset created by pairing claims from FEVER with relevant sentences, labeled as "Entailment" or "Contradiction" based on the FEVER label, and also including "Neutral" examples. This dataset was used for fine-tuning the RTE model.

Key Insights and Ideas

  • Combining Retrieval Strategies: The paper highlights the benefit of using both NER and TF-IDF similarity for retrieving relevant documents, suggesting that different types of information are captured by each method.
  • Importance of Fine-tuning for Entailment: General-purpose RTE models trained on datasets like SNLI may not generalize well to the specific domain of fact verification. Fine-tuning on a domain-specific dataset (FEVER SNLI-style) significantly improved performance.
  • Feature Engineering from Probabilities: Deriving rich features from the output probabilities of the RTE module is crucial for the final classification. The paper explores various ways to quantify the support and strength of evidence.
  • Limitations of Sentence-Level Entailment: The RTE module's limitation to pairwise sentence comparison means that claims requiring synthesis of information from multiple sentences are challenging to verify accurately.

Key Results

The DeFactoNLP system achieved the following scores on the FEVER 2018 Shared Task:

  • Evidence F1-score: 0.4277
  • Label Accuracy: 0.5136
  • FEVER Score: 0.3833

These results placed DeFactoNLP among the top-performing systems, with the 5th best evidence F1 score, 11th best label accuracy, and 12th best FEVER score out of 24 participating systems. The evidence F1 score was significantly better than the baseline system.

Limitations and Future Work

The authors acknowledge several limitations:

  • Low Evidence Recall: The primitive methods for retrieving candidate documents and sentences contributed to low evidence recall.
  • Limited RTE Capability: The pairwise sentence entailment approach struggles with claims requiring multi-sentence reasoning.
  • Classifier Reliability: The label accuracy was only marginally better than the baseline, suggesting the final classifier could be more robust.

Future work directions include:

  • Exploring more sophisticated methods for combining information from the RTE module.
  • Investigating triple extraction-based methods to improve sentence retrieval and identify more precise evidential sentences.
  • Employing entity linking for named entity disambiguation to avoid retrieving information from unrelated Wikipedia pages.
  • Incorporating coreference resolution to better handle sentences that use pronouns or referential expressions.

Summary updated

BibTeX
@inproceedings{reddy2018defactonlp,
  author = {Aniketh Janardhan Reddy and Gil Rocha and Diego Esteves},
  title = {{DeFactoNLP: Fact Verification using Entity Recognition, TFIDF Vector Comparison and Decomposable Attention}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure