Abstract
Automated fact extraction and verification is a challenging task that involves finding relevant evidence sentences from a reliable corpus to verify the truthfulness of a claim. Existing models either (i) concatenate all the evidence sentences, leading to the inclusion of redundant and noisy information; or (ii) process each claim-evidence sentence pair separately and aggregate all of them later, missing the early combination of related sentences for more accurate claim verification. Unlike the prior works, in this paper, we propose Hierarchical Evidence Set Modeling (HESM), a framework to extract evidence sets (each of which may contain multiple evidence sentences), and verify a claim to be supported, refuted or not enough info, by encoding and attending the claim and evidence sets at different levels of hierarchy. Our experimental results show that HESM outperforms 7 state-of-the-art methods for fact extraction and claim verification. Our source code is available at https://github.com/ShyamSubramanian/HESM.
Introduction
In the era of rapid information dissemination, automated Fact Extraction and VERification (FEVER) has become a critical tool for identifying misinformation. The task requires retrieving evidence from reliable sources, such as Wikipedia, to determine if a claim is Supported, Refuted, or if there is Not Enough Info.
Researchers identify a "Goldilocks" problem in existing models: some concatenate all potential evidence, introducing significant noise, while others process sentences individually, failing to capture "multi-hop" claims where the truth only emerges when specific sentences are read together. HESM bridges this gap by identifying and modeling evidence sets—groups of semantically linked sentences—and evaluating them at multiple levels of hierarchy.
Methodology
The HESM framework consists of a three-stage pipeline designed to improve both retrieval recall and verification accuracy:
- Document Retriever: Uses keyword matching and a Neural Semantic Matching Network (NSMN) to identify the top 10 relevant Wikipedia articles.
- Multi-hop Evidence Retriever: This component uses two iterations. The first finds the most relevant sentences, while the second follows hyperlinks within those sentences to find "hidden" evidence in other documents. This iterative strategy boosted recall from 0.868 to 0.905 on the FEVER dataset.
- Claim Verification: This core module processes each evidence set individually using an ALBERT-based modeling block. It then employs a Hierarchical Aggregator with two parallel paths:
- Non-contextual Aggregation: Combines individual verdicts, effective for simple claims.
- Contextual Aggregation: Uses a Transformer encoder to allow different evidence sets to interact, which is crucial for complex claims where evidence is scattered across different sets.
Results and Discussion
HESM was evaluated on the FEVER dataset, a benchmark containing over 185,000 human-annotated claims. The model outperformed seven state-of-the-art baselines:
- FEVER Score: Achieved 71.48% with ALBERT-Large, surpassing previous high-water marks like KGAT (70.38%).
- Label Accuracy: Reached 74.64%, demonstrating improved reasoning over retrieved text.
Key insights from the study suggest that evidence structure is paramount. Approximately 16.82% of FEVER claims require multiple sentences, and 12.5% require sentences from different documents. By treating evidence as sets rather than flat lists, HESM captures the semantic links necessary for human-like fact-checking. Furthermore, the hybrid aggregation strategy ensures the model remains efficient for simple tasks while remaining powerful enough for complex ones.
Conclusion
HESM represents a significant advancement in automated fact-checking by introducing hierarchical evidence set modeling. While the current model relies on hyperlinks for multi-hop retrieval, future work may explore scaling this logic to even more complex claims requiring three or more "hops" and adapting the framework for less structured environments beyond Wikipedia.
Summary updated