Team Papelo: Transformer Networks at FEVER

Christopher MalonEMNLP 2018

TL;DR

This paper introduces a novel system for fact verification that leverages Transformer networks, pre-trained on language modeling, to achieve state-of-the-art performance on the FEVER dataset. By improving both evidence retrieval and the precision of the entailment classifier, their system significantly enhances the ability to determine if a claim is supported, refuted, or undecided based on provided evidence.

What Problem Does This Paper Address?

The FEVER (Fact Extraction and VERification) dataset presents a significant challenge in natural language processing, requiring systems to not only retrieve relevant evidence from Wikipedia but also to accurately determine the relationship (entailment, refutation, or neutrality) between a given claim and the retrieved evidence. Existing methods often struggle with the abstract vocabulary, out-of-vocabulary terms, and the nuanced nature of evidence, especially when claims are refuted or require multiple pieces of evidence for verification. This paper aims to address these limitations by developing a more robust and precise entailment classifier and improving the evidence retrieval process.

Methods

The core of the proposed system is an entailment module based on a Transformer network. Unlike previous approaches that might concatenate multiple premise sentences, this system evaluates each premise statement individually against the claim. The Transformer network, pre-trained for language modeling (specifically, OpenAI's GPT model), is fine-tuned on the FEVER dataset. This architecture naturally handles out-of-vocabulary words and offers superior performance compared to other models like Decomposable Attention and ESIM.

To improve evidence retrieval, the system goes beyond simple TF-IDF matching. It incorporates:

  • Title-based retrieval: Adding the Wikipedia page title to premise statements when computing TF-IDF scores.
  • Named Entity Recognition (NER): Retrieving additional articles based on named entities and capitalized phrases found in the claim.
  • Film-specific retrieval: A heuristic to retrieve pages titled "X (film)" when "X" is identified as a potential entity.
  • Article-level retrieval: Expanding the search from just the top five sentences to entire articles, significantly increasing the pool of potential evidence.

Datasets

  • FEVER: The primary dataset used for training and evaluation. The authors introduce a new benchmark called "FEVER One" where premise statements are evaluated individually, contrasting with the baseline's "FEVER Five" which concatenates multiple sentences. They also create "FEVER Title One" by including the Wikipedia page title with each premise.
  • SNLI (Stanford Natural Language Inference): Mentioned as a basis for comparison, but noted that FEVER premises differ significantly in length and vocabulary.
  • BookCorpus: Used for the initial pre-training of the Transformer network's language modeling branch.

Key Insights and Ideas

  • Transformer's Superiority: The Transformer network, particularly when pre-trained on language modeling, significantly outperforms other entailment models like ESIM and Decomposable Attention on the FEVER task.
  • Individual Premise Evaluation: Evaluating each premise statement separately, rather than concatenating them, leads to better performance, especially when combined with the Transformer's precision.
  • The Power of Titles: Including the Wikipedia page title as part of the premise statement for TF-IDF calculation provides a substantial boost to evidence retrieval.
  • NER for Broader Retrieval: Leveraging named entity recognition to retrieve relevant articles beyond direct keyword matches is crucial for finding evidence, especially for refuted claims.
  • Balancing Retrieval and Entailment: The paper highlights the interplay between retrieval and entailment. While expanding retrieval significantly increases the evidence pool, the precision of the entailment classifier is key to filtering this information effectively.

Key Results

The system achieves competitive results on the FEVER shared task test set:

  • FEVER Score: .5736
  • Label Accuracy: .6108
  • Evidence F1: .6485

In preliminary standings, their system ranked fourth in FEVER score and first in evidence F1. The paper also shows significant improvements in evidence retrieval rates, reaching 90.1% when considering entire articles.

Limitations and Future Work

  • Handling Multi-Evidence Claims: The current system primarily focuses on single-evidence claims due to the complexity of aggregating decisions from multiple supporting or refuting statements. The authors acknowledge that a small fraction of FEVER claims require multiple pieces of evidence.
  • Pronoun Resolution: The system's awareness of pronoun antecedents is limited. While adding the page title helps, a dedicated coreference resolution system could further improve performance but would add computational cost.
  • Dynamic Retrieval: The authors suggest that future work could explore dynamic retrieval strategies, where the system decides to seek more information based on the entailment process itself, rather than relying on a fixed set of retrieved documents.

Summary updated

BibTeX
@inproceedings{malon2018team,
  author = {Christopher Malon},
  title = {{Team Papelo: Transformer Networks at FEVER}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}

Loading related content...

Figure