Abstract
We contribute a new dataset for the task of automated fact checking and an evaluation of state of the art algorithms. The dataset includes claims (from speeches, interviews, social media and news articles), review articles published by professional fact checkers and premise articles used by those professional fact checkers to support their review and verify the veracity of the claims. An important challenge in the use of premise articles is the identification of relevant passages that will help to infer the veracity of a claim. We show that transferring a dense passage retrieval model trained with review articles improves the retrieval quality of passages in premise articles. We report results for the prediction of claim veracity by inference from premise articles.
Introduction: Beyond Claim Entailment
Most automated fact-checking research currently treats the problem as a "claim entailment" task: given a claim and a summary written by a professional fact-checker, can the AI determine if the summary supports the claim? While useful, this does not reflect the actual work fact-checkers do. The real challenge is claim inference, where a fact-checker must sift through multiple, often long, source articles (premises) to find relevant facts and then synthesize those facts to reach a verdict.
Researchers from the University of Waterloo have introduced WatClaimCheck, a massive new dataset designed to move automated fact-checking closer to these real-world applications. By providing the raw "premise articles" used by professionals, this dataset challenges AI to find and synthesize evidence from scratch.
The WatClaimCheck Dataset
WatClaimCheck consists of 33,697 claims spanning from December 1996 to July 2021, aggregated from eight major fact-checking organizations such as PolitiFact, Snopes, and AFP Fact Check. Unlike existing benchmarks like FEVER (which uses Wikipedia) or PubHealth (which is domain-specific), WatClaimCheck provides raw, multi-domain premise articles for a large volume of real-world claims.
Methodology: A Two-Stage Pipeline
The authors propose a two-stage pipeline to tackle the complexity of claim inference:
- Evidence Sentence Extraction: Since premise articles are long, the system must identify relevant sentences. The authors compared standard keyword matching (TF-IDF) against a novel adaptation of Dense Passage Retrieval (DPR). They used a transfer learning approach, training the DPR model using professional review articles as "gold" evidence to learn what relevant information looks like, then transferred that knowledge to retrieve sentences from raw premise articles.
- Claim Veracity Inference: Extracted sentences are fed into a classifier to predict a verdict (True, False, or Partially True/False). Tested architectures included Bi-LSTMs, Hierarchical Attention Networks (HAN), and RoBERTa.
Key Insights and Results
The "Review" Shortcut
A significant finding was that predicting a verdict from a professional review article is much easier (Macro F1 of ~0.82) than predicting it from raw source articles (~0.58). This highlights that current AI is proficient at "reading the cliff notes" but struggles with the actual research process.
The Temporal Trap
Most NLP research uses random splits for training and testing, which is unrealistic for the "streaming" nature of news. The authors introduced prequential evaluation—testing the model on 6-month windows of time. They discovered that performance drops significantly when the model encounters new topics, such as the sudden surge of COVID-19 claims, proving that models struggle with distribution shifts over time.
Performance Metrics
- Retrieval: The DPR method significantly outperformed TF-IDF, achieving a Top-100 recall of 54% compared to TF-IDF's 35%.
- Veracity Prediction: The best-performing model was a pooled RoBERTa-base approach, which achieved a Macro F1 score of 0.580 using DPR-retrieved evidence.
Conclusion
WatClaimCheck sets a new "north star" for automated fact-checking by focusing on raw evidence rather than summaries. Future work must address the "black box" nature of current models to provide explainable conclusions and improve zero-shot learning to adapt to the rapidly changing news cycle without constant retraining.
Summary updated