TL;DR
This paper investigates whether current automatic fake news detection models truly "reason" by combining claims with evidence, or if they simply exploit inherent signals within the evidence itself. Surprisingly, they find that models often perform best when ignoring the claim, suggesting a potential flaw in how evidence is currently utilized and raising questions about genuine reasoning capabilities.
What Problem Does This Paper Address?
The proliferation of misinformation online is a significant societal challenge. Automatic fact-checking systems aim to combat this by assessing the veracity of claims, typically by using external evidence. The prevailing assumption is that these models learn to "reason" by jointly processing a claim and its supporting or refuting evidence. However, this paper questions whether this is truly happening. They hypothesize that if models are genuinely reasoning, combining both claim and evidence should lead to better performance than using either alone. If models perform well using only evidence, it suggests they might be exploiting a shortcut or bias within the evidence itself, rather than engaging in true inferential reasoning.
Methods
The authors evaluate three types of fact-checking models with varying complexity:
- Term-frequency based Random Forest: A traditional machine learning approach using bag-of-words representations of concatenated claim and evidence.
- GloVe-based LSTM: A deep learning model using GloVe word embeddings and an attention-weighted bidirectional LSTM to encode claims and evidence snippets.
- BERT-based model: A state-of-the-art transformer model that uses BERT embeddings for claims and evidence snippets, with an attention mechanism to combine them.
For each model, they conduct experiments in three settings: using only the claim, using only the evidence, and using both claim and evidence. They also explore the impact of evidence by systematically removing snippets from the top (most relevant) and bottom of the search results.
Datasets
The study uses two well-known political fact-checking datasets:
- PolitiFact: A dataset of claims and associated evidence.
- Snopes: Another dataset of claims and associated evidence.
These datasets were extracted from the MultiFC dataset. For evaluation, they use both within-dataset and out-of-dataset splits to assess generalizability.
Key Insights and Ideas
- Evidence is King (and often sufficient): The most striking finding is that models, particularly BERT, often achieve their best performance using only the evidence, even outperforming models that use both claim and evidence. This strongly suggests that the evidence itself contains a powerful predictive signal for claim veracity.
- Reasoning vs. Exploitation: The results challenge the assumption that models are learning to reason. Instead, they may be exploiting an inherent signal within the evidence that is correlated with the claim's truthfulness, without truly understanding the logical connection between the claim and the evidence.
- The Claim Can Be Harmful: In some cases, including the claim actually decreases the model's effectiveness, further supporting the idea that the claim might be introducing noise or irrelevant information compared to the strong signal in the evidence.
- Evidence Ranking Matters: Removing evidence snippets from the top of the search results (presumably the most relevant) leads to a larger performance drop than removing snippets from the bottom. This indicates that the order in which evidence is presented by search engines is meaningful and directly impacts fact-checking performance.
- Model Sensitivity to Evidence: While BERT shows a smaller performance drop when evidence is removed, traditional models like Random Forest and LSTM are more sensitive to the presence of evidence, dropping significantly when it's removed. This suggests different models might rely on evidence in fundamentally different ways.
Key Results
- BERT with Evidence Dominates: The BERT model trained solely on evidence achieved the best results in 4 out of 8 evaluation columns and the best macro F1 score in 3 out of 4 within- and out-of-dataset scenarios.
- Claim+Evidence Rarely Best: The combination of claim and evidence only marginally achieved the best results in one specific scenario (Snopes micro F1).
- Evidence Alone is Highly Predictive: For BERT, using only evidence achieved performance comparable to, or better than, using both claim and evidence.
- Evidence Ranking Impact: Figure 1 demonstrates that removing the top-ranked evidence snippets significantly degrades performance, while removing bottom-ranked snippets has a less severe impact.
Limitations and Future Work
- Problematic Evidence Gathering: The authors highlight a critical issue: the current method of evidence gathering (e.g., using Google search snippets) provides a strong signal that allows models to bypass the claim. This makes models overly reliant on the search system's underlying mechanisms, which could change and impact performance unpredictably.
- Fundamental Reasoning Gap: If models are not truly reasoning, they might fail on claims where the evidence is ambiguous or where the claim's veracity depends on nuanced understanding rather than direct factual correlation. For example, if the evidence is the same for "the earth is round" and "the earth is flat," a model solely reliant on evidence might struggle.
- Future Directions: The paper suggests a need to rethink how evidence is defined and utilized in automatic fact-checking. Future work could explore methods that explicitly encourage models to reason over the claim and evidence, or investigate alternative evidence retrieval strategies that are less prone to exploitation.
Summary updated