Abstract
Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) have demonstrated significant potential in automated fact-checking. However, existing methods face limitations in insufficient evidence utilization and lack of explicit verification criteria. Specifically, these approaches aggregate evidence for collective reasoning without independently analyzing each piece, hindering their ability to leverage the available information thoroughly. Additionally, they rely on simple prompts or few-shot learning for verification, which makes truthfulness judgments less reliable, especially for complex claims. To address these limitations, we propose a novel method to enhance evidence utilization and introduce explicit verification criteria, named EVICheck. Our approach independently reasons each evidence piece and synthesizes the results to enable more thorough exploration and enhance interpretability. Additionally, by incorporating fine-grained truthfulness criteria, we make the model's verification process more structured and reliable, especially when handling complex claims. Experimental results on the public RAWFC dataset demonstrate that EVICheck achieves state-of-the-art performance across all evaluation metrics. Our method demonstrates strong potential in fake news verification, significantly improving the accuracy.
Introduction
Automated fact-checking has recently pivoted toward using Large Language Models (LLMs) paired with Retrieval-Augmented Generation (RAG). However, two major bottlenecks persist: Insufficient Evidence Utilization (IEU) and a lack of explicit verification criteria. Most RAG systems collect a batch of evidence and ask the LLM to reason across the entire aggregate at once. This often leads to the model overlooking critical details or contradictions hidden in individual snippets, as stronger or more frequent evidence pieces "overshadow" subtler ones. Furthermore, many systems rely on simple prompts or few-shot examples, which struggle with nuanced "Half-True" claims or complex causal relationships.
Methodology: The EVICheck Framework
EVICheck introduces a two-module pipeline designed to mimic the structured reasoning of a human fact-checker by emphasizing independent analysis and fine-grained rubrics.
Module 1: Evidence Acquisition & Preliminary Reasoning
Instead of a single search, the system uses a multi-round inference loop. It extracts core elements of a claim, generates multiple verification questions, and selects the optimal query for a search API. Crucially, it performs independent reasoning on each retrieved piece of information, generating a preliminary conclusion and reasoning process for every snippet before synthesis. This ensures that no contradictory data point is ignored.
Module 2: Combined Verification
The system aggregates all preliminary reasoning results and applies Fine-grained Truthfulness Criteria. This specific set of rules defines what constitutes "True," "False," or "Half" (e.g., a claim is "False" if it contains a "Causal Relationship Error," even if individual events are true). The final judgment is processed by a fine-tuned Llama-3-8B-Instruct model, trained specifically to follow these criteria using Low-Rank Adaptation (LoRA).
Experimental Evaluation
The framework was evaluated on the RAWFC dataset, an English fake news benchmark derived from Snopes containing 2,012 entries with manually annotated "golden label" explanations.
Key Results
EVICheck outperformed existing state-of-the-art (SOTA) methods across all primary metrics. The combination of GPT-4 for reasoning and fine-tuned Llama-3 for verification reached an F1-score of 0.633, significantly higher than previous SOTA methods like CofCED (0.520) and RAFTS (0.573). Ablation studies showed that removing the "preliminary reasoning" step led to a 7.77% performance drop, confirming that independent analysis is a core driver of success.
Performance and Bias Analysis
A significant insight from the study is the "negative bias" observed in frontier models like GPT-4. Out-of-the-box, GPT-4 frequently misclassifies "Half-True" claims as "False." By fine-tuning a smaller model (Llama-3) on a specific truthfulness rubric, EVICheck achieved a more balanced and accurate judgment.
As illustrated in the confusion matrices, the fine-tuned Llama-3 model significantly improves the classification of "half" labels, whereas GPT-4 shows a heavy skew toward "false" predictions for those same instances.
Conclusion and Future Work
EVICheck demonstrates that independent evidence analysis and structured rubrics significantly improve fact-checking accuracy and interpretability. While highly effective on structured claims, the system currently faces limitations with informal social media content (slang, sarcasm) and multimodal misinformation. Future research will focus on integrating social-media-specific APIs and expanding the framework to handle image and video-based reasoning.
Summary updated