TL;DR
This paper introduces a novel Multi-Level Attention (MLA) model for evidence-based fact-checking, which efficiently verifies claims against textual evidence using a sequence-based approach. The MLA model achieves state-of-the-art results on the FEVER dataset by effectively capturing inter-sentence relationships and mitigating training-test discrepancies, outperforming complex graph-based methods.
What Problem Does This Paper Address?
The proliferation of false and misleading information online poses a significant challenge. Evidence-based fact-checking aims to combat this by verifying claims against supporting or refuting evidence from textual sources. A key difficulty lies in effectively representing the complex relationships between a claim and multiple pieces of evidence, especially when the evidence itself needs to be identified and selected from a larger corpus. While recent approaches have explored sophisticated graph-based neural networks, this paper argues that simpler sequence-based models, when properly designed, can be equally, if not more, effective. Furthermore, a critical challenge is the discrepancy between training data (which uses ground-truth evidence) and testing data (which relies on predicted evidence), leading to performance degradation.
Methods
The authors propose a Multi-Level Attention (MLA) model that operates on linear sequences, eschewing complex graph structures. The core of their approach involves:
- BERT-based Sentence Representation: They leverage BERT to obtain rich vector representations of claims and candidate evidence sentences, capturing token-level interactions.
- Multi-Level Attention: The model employs a multi-level attention mechanism that includes:
- Token-level self-attention: Applied to the concatenated claim and evidence sentences to capture fine-grained interactions.
- Sentence-level self-attention: Applied to the representations of individual evidence sentences to learn inter-sentence relationships.
- Cross-attention: Integrates the claim representation with the aggregated evidence representation.
- Mitigating Training-Test Discrepancies:
- Sentence Selection: They propose a method for training the sentence selection model using a mixture of ground-truth and predicted negative examples from retrieved documents, making it more robust to test-time scenarios.
- Veracity Prediction: For training the veracity prediction model, they use a mix of true and predicted evidence sentences, particularly to address the "NOTENOUGHINFO" class, which is often underspecified in training data.
- Joint Training: The sentence selection and veracity prediction models are jointly trained, allowing the sentence selection scores to act as a "gate" to modulate the evidence representation in the veracity prediction module.
Datasets
The primary dataset used is FEVER (Fact Extraction and VERification). The authors utilize the document retrieval results provided by Hanselowski et al. (2018) for their experiments. They do not introduce a new dataset.
Key Insights and Ideas
- Simplicity can be powerful: The paper demonstrates that a well-designed sequence-based model can outperform complex graph-based approaches for fact-checking, suggesting that graph structures might be overkill for this task.
- Bridging the training-test gap is crucial: The proposed methods for incorporating predicted examples into training significantly improve robustness and performance, highlighting the importance of aligning training and testing conditions.
- Joint training benefits both tasks: Leveraging sentence selection scores to guide veracity prediction leads to better overall performance, showing the synergy between these sub-tasks.
- Multi-level attention captures rich interactions: The hierarchical attention mechanism effectively models relationships at both token and sentence levels, contributing to the model's strong performance.
Key Results
The MLA model achieves state-of-the-art results on the FEVER dataset. Specifically, on the blind test set, the MLA model with RoBERTa-large yields 1.09% improvement in label accuracy (LA) and 1.42% improvement in FEVER score compared to the previous best published model (CorefRoBERTa). The paper also shows that their proposed sentence selection strategy, which uses a mixture of ground-truth and predicted examples, is crucial, with scores dropping by around 2% when this strategy is omitted.
Limitations and Future Work
- Limitations:
- The model's performance can be sensitive to the quality of the upstream document retrieval and sentence selection components.
- The paper acknowledges that some errors require complex reasoning that the current model struggles with (e.g., understanding numerical relationships).
- There are instances where the models disagree with annotations, suggesting potential issues with annotation consistency or the definition of "half-true" claims.
- Future Work:
- Exploring better upstream components (document retriever, sentence selector) to further enhance performance.
- Investigating adversarial examples to improve model robustness while maintaining high performance on normal inputs.
- Developing models that can better handle complex reasoning tasks.
Summary updated