Integrating Entity Linking and Evidence Ranking for Fact Extraction and Verification

Motoki Taniguchi, Tomoki Taniguchi, Takumi Takahashi, Yasuhide Miura, Tomoko OhkumaEMNLP 2018

TL;DR

This paper presents a system for fact extraction and verification, specifically tackling the FEVER task. Their approach integrates entity linking, sentence retrieval, and a neural network for recognizing textual entailment, achieving a FEVER score of 0.4016 and outperforming the baseline system by significantly improving evidence precision.

What Problem Does This Paper Address?

The internet is awash with textual information, creating a pressing need for automated systems that can extract and verify facts. The Fact Extraction and VERification (FEVER) task specifically addresses this by requiring systems to classify textual claims as SUPPORTED, REFUTED, or NOT ENOUGH INFO (NEI), and crucially, to provide supporting evidence from Wikipedia for claims that are not NEI. The challenge lies in efficiently and accurately identifying relevant documents and sentences from a vast corpus like Wikipedia that can either support or refute a given claim.

Methods

The authors propose a pipeline system composed of three main components:

  • Document Selection: This component identifies relevant Wikipedia documents for a given claim. They employ a simple but effective entity linking approach that uses exact text matching between entities mentioned in the claim and Wikipedia page titles. This prioritizes precision over recall, aiming to select only highly relevant documents.
  • Sentence Retrieval: Similar to the baseline system, this component retrieves candidate evidence sentences from the selected documents. They use a TF-IDF based approach, selecting the K nearest sentences to the claim based on cosine similarity. They found K=15 to be optimal.
  • Recognizing Textual Entailment (RTE): This is the core of their novel contribution. They utilize the DEISTE model, a state-of-the-art neural network for inter-sentence interactions. This component simultaneously ranks candidate sentences and classifies the claim (SUPPORTED, REFUTED, or NEI). A key aspect is their strategy for handling NEI claims: if the model cannot predict SUPPORTED or REFUTED with high confidence, it defaults to NEI. They also introduce a threshold (Pt=0.93) to control this confidence level.

Datasets

  • Training: The official FEVER training dataset was used to train the RTE component.
  • Development and Test: The development and test datasets from Thorne et al. (2018) were used for parameter tuning and performance evaluation.

Key Insights and Ideas

  • Precision-focused Document Selection: By prioritizing exact text matches for entity linking, the system achieves higher precision in document selection, leading to more reliable evidence. While this might reduce recall, the authors argue it's a worthwhile trade-off for this task.
  • Joint Ranking and Classification: The neural network approach that simultaneously ranks evidence sentences and classifies the claim is a significant advancement. This allows the model to learn the interplay between evidence quality and claim veracity more effectively.
  • Confidence-based NEI Prediction: The strategy of predicting NEI when confidence in SUPPORTED/REFUTED is low is a pragmatic approach to handle uncertainty and improve overall system robustness.

Key Results

The system achieved a FEVER score of 0.4016 on the test dataset, outperforming the baseline system. Notably, their system showed a significant improvement in evidence precision (0.6986 vs. 0.1084). While evidence recall decreased, the overall evidence F1 score increased by 17 points compared to the baseline. Their submission was ranked 9th in the shared task.

Limitations and Future Work

The authors acknowledge that their document selection component, while precise, was "too strict," leading to degraded evidence recall. As a future direction, they plan to explore more sophisticated entity linking approaches to improve recall without sacrificing too much precision.

Summary updated

BibTeX
@inproceedings{taniguchi2018integrating,
  author = {Motoki Taniguchi and Tomoki Taniguchi and Takumi Takahashi and Yasuhide Miura and Tomoko Ohkuma},
  title = {{Integrating Entity Linking and Evidence Ranking for Fact Extraction and Verification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure