TL;DR
This paper presents a system for claim verification that leverages semantic frames from FrameNet to improve the retrieval of relevant evidence sentences from Wikipedia. By combining this linguistically-informed retrieval with a simple multi-layer perceptron classifier, the UMBC team achieved a competitive score in the FEVER shared task, demonstrating the effectiveness of semantic frames for this challenging task.
What Problem Does This Paper Address?
Fact verification, especially at scale, is a critical challenge in the age of information overload. The FEVER (Fact Extraction and VERification) shared task aims to automate this process by taking a factual claim and determining if it is supported, refuted, or if there's not enough information to decide, based on evidence found in Wikipedia. The core difficulty lies in two main areas: first, efficiently and accurately retrieving the most relevant sentences from a vast corpus like Wikipedia that can serve as evidence, and second, classifying the claim based on this evidence. Traditional approaches often rely on keyword matching or simpler entity-based retrieval, which can miss nuanced relationships and semantic connections crucial for accurate verification.
Methods
The UMBC team's approach tackles the FEVER problem with a two-pronged strategy:
- Semantic Frame-Based Information Retrieval: Instead of relying solely on keywords, they employ semantic frames from FrameNet. A frame represents a situation or event and its participants (e.g., "Commerce_buy" frame with roles like Buyer, Seller, Goods). Claims and Wikipedia sentences are annotated with these frames. Evidence retrieval then involves matching frames evoked by the claim with frames found in Wikipedia sentences. This allows for a more semantically rich matching, going beyond simple word overlap. They experimented with different retrieval strategies, including using only document retrieval, sentence retrieval, and a combination, as well as incorporating introduction sections of Wikipedia pages.
- Multi-Layer Perceptron (MLP) Classification: For classifying the claim as Supported, Refuted, or NotEnoughInfo, they used a relatively simple two-layer MLP. Features for the classifier were derived from word embeddings of the claim, averaged over its length. They experimented with various MLP configurations (learning rate, hidden layer size, dropout, etc.) to find optimal settings.
Datasets
- FEVER Dataset: This is the primary dataset used for training and evaluation, consisting of claims and corresponding Wikipedia evidence sentences. The paper notes a discrepancy between the Wikipedia dump used for the FEVER dataset (2017) and the one they used for frame annotation (2016), which introduced some challenges.
- FrameNet: Used for its rich semantic frame annotations.
- Wikipedia: The corpus from which evidence sentences are retrieved.
Key Insights and Ideas
- Semantic Frames Enhance Retrieval: The core idea is that semantic frames capture deeper meaning than keywords. Matching frames between claims and potential evidence sentences significantly improves the quality and relevance of retrieved evidence, as demonstrated by their strong performance on the information retrieval task.
- Frame Annotation Imperfection: The frame annotator is not perfect, leading to situations where claims cannot be matched with evidence, even if it exists. This highlights the importance of robust annotation pipelines and potentially more forgiving matching mechanisms.
- Simple Classifier Can Be Effective with Good Evidence: Despite using a relatively simple MLP classifier, their system achieved good results. This suggests that the quality of the retrieved evidence is a dominant factor in the overall performance of claim verification systems.
- Wikipedia Version Mismatches Matter: The difference in Wikipedia versions between the FEVER dataset and their annotated corpus caused issues, particularly in aligning sentences and potentially impacting classification accuracy.
Key Results
- Information Retrieval Improvement: Their system achieved a relative improvement of 131% on retrieving evidence over a baseline, scoring 10th out of 24 teams in the information retrieval task of the FEVER shared task.
- Overall Performance: Their best submission achieved an Evidence F1 score of 0.3966 and an overall FEVER score of 0.2628 F1 points.
- Classification Performance: While their classification performance was lower than some top-performing systems that used more sophisticated approaches, their system demonstrated that a simpler classifier can be effective when paired with high-quality evidence.
- Impact of Retrieval Strategies: Different retrieval strategies (NFC, NFUC, NFIC) showed varying performance, with NFUC (union of frame-based and introduction-based retrieval) performing best among their tested settings.
Limitations and Future Work
- Frame Annotator Accuracy: The imperfection of the automatic frame annotator is a significant limitation, leading to missed evidence and impacting overall performance.
- Wikipedia Version Mismatch: The difference in Wikipedia dumps used for the dataset and for annotation created challenges in sentence alignment and potentially affected results.
- Future Work:
- Integrate with Knowledge Graphs: The authors plan to integrate their approach into a larger system for cleaning noisy knowledge graphs, leveraging semantic frames for better entity and relation understanding.
- Improve Entity Linking and Coreference: Enhancing entity coreference resolution and linking to knowledge graphs (like Freebase, DBpedia, Wikidata) using tools like Kelvin could further improve performance.
- Exploit Relational Reasoning: Leveraging the ability of semantic frames to capture entailments, temporal ordering, causality, and generalization could lead to more sophisticated reasoning capabilities.
- Unified Wikipedia Versions: Using identical Wikipedia versions for datasets and annotations is crucial for future improvements.
Summary updated