TL;DR
This paper presents a novel fact-checking system for the FEVER challenge, which aims to verify claims using information from Wikipedia. Their system achieves a significant 25.5% improvement over the baseline by employing a two-part approach: first, retrieving relevant candidate sentences from Wikipedia, and second, using a modified decomposable attention network to determine if these sentences support, refute, or provide insufficient information to verify the claim.
What Problem Does This Paper Address?
The FEVER (Fact Extraction and VERification) challenge tackles the crucial problem of automated fact-checking. Given a claim, the goal is to find supporting or refuting evidence within a large corpus (Wikipedia) and then classify the claim as "SUPPORTS," "REFUTES," or "NOT ENOUGH INFO." This is a complex task because it requires not only retrieving relevant information but also understanding the nuanced relationship between a claim and potential evidence, often involving natural language inference. Existing baseline systems provide a starting point, but there's a clear need for improved accuracy and recall in both evidence retrieval and claim verification.
Methods
The Uni-DUE team's system follows a two-part architecture, similar to the baseline, but with key innovations:
- Part 1: Finding Candidate Sentences: This part focuses on mimicking human information retrieval.
- Keyword Extraction: They use a combination of Named Entity Recognition (NER), Constituency Parsing, and Dependency Parsing to extract keywords (named entities, NP-tagged phrases, subject/object phrases) from claims. This multi-pronged approach aims to capture a richer set of relevant terms than relying on a single method.
- Wikipedia Indexing: The Wikipedia dump is indexed using Lucene, with each sentence treated as a separate document. Crucially, they replace pronouns (He, She, It, They) with the Wikipedia page title to provide context when searching individual sentences.
- Multi-Type Queries: They employ three types of queries to search for candidate sentences within Wikipedia page titles and content, balancing precision and recall.
- Part 2: Making the Prediction: This part refines the candidate sentences and makes the final classification.
- Modified Decomposable Attention Network: They adapt the Decomposable Attention network (a standard for natural language inference) to predict textual entailment between a claim and each candidate sentence.
- Handling Long Sentences: A novel addition is a 2D convolutional layer applied to the attention matrix of longer sentences. This aims to better capture multiple statements within a single long sentence, improving the network's ability to discern relevant evidence.
- Merge Sentences: They also merge all candidate sentences into a single block of text and predict entailment, which proved particularly effective for identifying "REFUTES" claims.
- POS-Tag Scoring: A point system is introduced, assigning scores to candidate sentences based on the presence of specific Part-of-Speech (POS) tags (verbs, nouns, adjectives, adverbs, numbers) in both the claim and the sentence. This helps filter out less relevant sentences.
- Final Prediction: The system combines the entailment probabilities from individual sentence-claim pairs and the merged sentence block, along with the POS-tag scores, to make the final classification and select the top evidence sentences.
Datasets
- FEVER Dataset: This is the primary dataset for the challenge, consisting of annotated claims and evidence. It's split into training, development, and test sets.
- Wikipedia Dump: A large corpus of Wikipedia pages used for retrieving evidence.
- SNLI Dataset: Used in conjunction with the FEVER training set for training the modified Decomposable Attention network.
Key Insights and Ideas
- Keyword Extraction is Key: The multi-method approach to keyword extraction significantly improves the chances of finding relevant Wikipedia pages and, consequently, candidate sentences.
- Contextualizing Sentences: Replacing pronouns with page titles during indexing is a clever way to inject crucial context that would otherwise be lost when processing sentences in isolation.
- Decomposable Attention is Powerful, but Needs Augmentation: While the Decomposable Attention network is a strong foundation for entailment, the paper demonstrates that modifications like the 2D convolution for long sentences and the "merge sentences" approach are vital for improving performance on the FEVER task.
- POS-Tags as a Filter: The POS-tag scoring system acts as an effective heuristic to filter out noisy or irrelevant candidate sentences, leading to a substantial improvement in the overall score.
- "REFUTES" are Easier to Spot: The paper notes that their system, and the merge sentences approach specifically, is particularly good at identifying "REFUTES" claims, suggesting an asymmetry in the difficulty of identifying different claim types.
Key Results
- FEVER Score Improvement: The system achieved a FEVER score of 0.3927 on the development set, which is a 25.5% improvement over the baseline score of 0.3127.
- Evidence Recall: The candidate sentences found in the first part of their system included the actual evidence for 77.83% of the claims, significantly outperforming the baseline's 44.22%.
- Sentence Filtering: The POS-tag point system effectively filters out non-evidence sentences, with a significant percentage of sentences scoring 11 or less.
- Contribution of Features: The point system using POS-Tags provided the biggest improvement to the final score.
Limitations and Future Work
- Limitations:
- The paper notes that the highest incorrect predictions are for the "NOT ENOUGH INFO" label.
- The POS-tagging process for claims and candidate sentences was parallelized to speed it up, indicating that the original Stanford POS-Tagger was a bottleneck.
- Future Work:
- Investigate alternative methods for obtaining higher recall in the first part (evidence retrieval).
- Further improve the textual entailment component to reduce noise.
Summary updated