EVICheck: Evidence-Driven Independent Reasoning and Combined Verification Method for Fact-Checking

Lingxiao Wang, Lei Shi, Feifei Kou, Ligu Zhu, Chen Ma, Pengfei Zhang, Mingying Xu, Zeyu LiIJCAI 2025 · OpenReview

Abstract

Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) have demonstrated significant potential in automated fact-checking. However, existing methods face limitations in insufficient evidence utilization and lack of explicit verification criteria. Specifically, these approaches aggregate evidence for collective reasoning without independently analyzing each piece, hindering their ability to leverage the available information thoroughly. Additionally, they rely on simple prompts or few-shot learning for verification, which makes truthfulness judgments less reliable, especially for complex claims. To address these limitations, we propose a novel method to enhance evidence utilization and introduce explicit verification criteria, named EVICheck. Our approach independently reasons each evidence piece and synthesizes the results to enable more thorough exploration and enhance interpretability. Additionally, by incorporating fine-grained truthfulness criteria, we make the model's verification process more structured and reliable, especially when handling complex claims. Experimental results on the public RAWFC dataset demonstrate that EVICheck achieves state-of-the-art performance across all evaluation metrics. Our method demonstrates strong potential in fake news verification, significantly improving the accuracy.

Introduction

Automated fact-checking has recently pivoted toward using Large Language Models (LLMs) paired with Retrieval-Augmented Generation (RAG). However, two major bottlenecks persist: Insufficient Evidence Utilization (IEU) and a lack of explicit verification criteria. Most RAG systems collect a batch of evidence and ask the LLM to reason across the entire aggregate at once. This often leads to the model overlooking critical details or contradictions hidden in individual snippets, as stronger or more frequent evidence pieces "overshadow" subtler ones. Furthermore, many systems rely on simple prompts or few-shot examples, which struggle with nuanced "Half-True" claims or complex causal relationships.

Methodology: The EVICheck Framework

EVICheck introduces a two-module pipeline designed to mimic the structured reasoning of a human fact-checker by emphasizing independent analysis and fine-grained rubrics.

Comparison of EVICheck against traditional RAG-based fact-checking methods.

Module 1: Evidence Acquisition & Preliminary Reasoning

Instead of a single search, the system uses a multi-round inference loop. It extracts core elements of a claim, generates multiple verification questions, and selects the optimal query for a search API. Crucially, it performs independent reasoning on each retrieved piece of information, generating a preliminary conclusion and reasoning process for every snippet before synthesis. This ensures that no contradictory data point is ignored.

Module 2: Combined Verification

The system aggregates all preliminary reasoning results and applies Fine-grained Truthfulness Criteria. This specific set of rules defines what constitutes "True," "False," or "Half" (e.g., a claim is "False" if it contains a "Causal Relationship Error," even if individual events are true). The final judgment is processed by a fine-tuned Llama-3-8B-Instruct model, trained specifically to follow these criteria using Low-Rank Adaptation (LoRA).

Experimental Evaluation

The framework was evaluated on the RAWFC dataset, an English fake news benchmark derived from Snopes containing 2,012 entries with manually annotated "golden label" explanations.

Key Results

EVICheck outperformed existing state-of-the-art (SOTA) methods across all primary metrics. The combination of GPT-4 for reasoning and fine-tuned Llama-3 for verification reached an F1-score of 0.633, significantly higher than previous SOTA methods like CofCED (0.520) and RAFTS (0.573). Ablation studies showed that removing the "preliminary reasoning" step led to a 7.77% performance drop, confirming that independent analysis is a core driver of success.

Performance and Bias Analysis

A significant insight from the study is the "negative bias" observed in frontier models like GPT-4. Out-of-the-box, GPT-4 frequently misclassifies "Half-True" claims as "False." By fine-tuning a smaller model (Llama-3) on a specific truthfulness rubric, EVICheck achieved a more balanced and accurate judgment.

Confusion matrices for GPT-4 and the fine-tuned Llama-3 model, highlighting the reduction in negative bias.

As illustrated in the confusion matrices, the fine-tuned Llama-3 model significantly improves the classification of "half" labels, whereas GPT-4 shows a heavy skew toward "false" predictions for those same instances.

Conclusion and Future Work

EVICheck demonstrates that independent evidence analysis and structured rubrics significantly improve fact-checking accuracy and interpretability. While highly effective on structured claims, the system currently faces limitations with informal social media content (slang, sarcasm) and multimodal misinformation. Future research will focus on integrating social-media-specific APIs and expanding the framework to handle image and video-based reasoning.

Summary updated

Abstract

Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) have demonstrated significant potential in automated fact-checking. However, existing methods face limitations in insufficient evidence utilization and lack of explicit verification criteria. Specifically, these approaches aggregate evidence for collective reasoning without independently analyzing each piece, hindering their ability to leverage the available information thoroughly. Additionally, they rely on simple prompts or few-shot learning for verification, which makes truthfulness judgments less reliable, especially for complex claims. To address these limitations, we propose a novel method to enhance evidence utilization and introduce explicit verification criteria, named EVICheck. Our approach independently reasons each evidence piece and synthesizes the results to enable more thorough exploration and enhance interpretability. Additionally, by incorporating fine-grained truthfulness criteria, we make the model's verification process more structured and reliable, especially when handling complex claims. Experimental results on the public RAWFC dataset demonstrate that EVICheck achieves state-of-the-art performance across all evaluation metrics. Our method demonstrates strong potential in fake news verification, significantly improving the accuracy.
BibTeX
@inproceedings{wang2025evicheck,
  author = {Lingxiao Wang and Lei Shi and Feifei Kou and Ligu Zhu and Chen Ma and Pengfei Zhang and Mingying Xu and Zeyu Li},
  title = {{EVICheck: Evidence-Driven Independent Reasoning and Combined Verification Method for Fact-Checking}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=VYVoOFQmD4},
}
Figure