TL;DR
This paper introduces FarFetched, a novel framework for automated claim validation that leverages entity-centric reasoning and aggregated evidence from online news sources. It addresses the challenge of verifying claims in low-resource languages like Greek by building a graph of interconnected information and using semantic similarity and natural language inference.
What Problem Does This Paper Address?
In today's information-saturated world, the ability to quickly and accurately verify claims is crucial. However, this task is particularly challenging for low-resource languages due to a lack of robust NLP tools and datasets. Existing fact-checking systems often rely on curated sources or manual annotation, which are not scalable. FarFetched aims to bridge this gap by providing an automated, evidence-aware claim validation system that can be adapted to languages with limited linguistic resources, using Greek as a case study.
Methods
FarFetched employs a modular, multi-stage approach:
- News Collection: A multilingual crawler gathers news articles from various sources.
- Graph Database Population: Collected articles are stored in a graph database, where articles and their sentences are nodes, linked by relationships.
- Entity Linking: This crucial step identifies named entities within claims and news articles, linking them to Wikipedia concepts (via wikification and PageRank) to create a knowledge graph. This helps uncover latent connections between different pieces of information.
- Evidence Constructor: Based on the identified entities in a user's claim, the system queries the graph database to find shortest paths connecting these entities through article sections. These sections are then aggregated to form candidate evidence sequences.
- Semantic Textual Similarity (STS): To select the most relevant evidence, a Greek-specific STS model is trained using a multilingual knowledge distillation approach. This model compares the user's claim with candidate evidence sequences to find the best match.
- Natural Language Inference (NLI): Finally, the selected best evidence (premise) and the user's claim (hypothesis) are fed into a fine-tuned Greek NLI model to determine if the claim is entailed, contradicted, or neutral.
Datasets
- FEVER: A subset of the FEVER dataset was manually translated into Greek for evaluating the end-to-end performance of FarFetched.
- XNLI: The Greek subset of the XNLI dataset was used to benchmark the trained NLI model.
- STS2017: The test subset of the STS2017 benchmark was used to evaluate the STS model, with a cross-lingual English-Greek version manually created.
- AllNLI: The combined SNLI and MultiNLI corpora were used to train the Greek NLI model, with English-to-Greek machine translation applied.
Key Insights and Ideas
- Entity-centric reasoning is key: By focusing on entities as connectors, FarFetched can uncover relationships between disparate pieces of information, even across different articles.
- Bridging the low-resource language gap: The framework's modularity and the development of specific Greek STS and NLI models demonstrate a viable path for adapting advanced NLP techniques to languages with limited resources.
- Dynamic evidence aggregation: The system's ability to continuously ingest new information means that claim validation is not static; verdicts can shift as new evidence emerges.
- Interpretable reasoning: While not a primary focus, the approach provides a degree of interpretability by showing the evidence used to support or refute a claim.
Key Results
- End-to-end performance: On the Greek FEVER subset, FarFetched achieved an overall accuracy of 0.73, with balanced precision and recall for "REFUTES" and "SUPPORTS" classes. This shows a significant improvement over baseline models.
- STS performance: The trained Greek STS model achieved strong results on the EN-EL version of STS2017, outperforming a comparable multilingual model.
- NLI performance: The trained Greek NLI model achieved an F1-score of 78.3 on the Greek XNLI test set, performing on par with or better than existing multilingual and monolingual Greek models.
- Qualitative examples: The paper demonstrates scenarios where FarFetched correctly identifies contradicting claims and adapts its verdict based on newly incorporated information, highlighting the system's dynamic nature.
Limitations and Future Work
- Entity Linking Ambiguity: Highly ambiguous entities and name variations can lead to incorrect evidence selection and "neutral" NLI verdicts. The wikification module's sensitivity is tunable, creating a trade-off between precision and recall.
- Evidence Construction Strategy: The current reliance on shortest paths for evidence construction can be computationally intensive for claims with many entities. The system doesn't distinguish between opinions and facts, nor does it perform temporal evaluation of evidence.
- NLI Temporal Reasoning: The NLI model's ability to reason based solely on temporal and aspectual properties of predicates is limited. Claims heavily reliant on these nuances might be challenging.
Future work includes improving entity linking robustness, developing more sophisticated evidence construction strategies (e.g., exploring top-n candidates), and addressing the temporal and factual distinction limitations. The authors also plan to extend the framework to other languages.
Summary updated