Introduction
Verifying scientific claims is an increasingly complex challenge due to the exponential growth of research literature. While Large Language Models (LLMs) offer potential assistance, their application in scientific contexts is hindered by hallucinations and a lack of traceability. Furthermore, scientific truth is often non-binary, requiring a nuanced understanding of underlying assumptions that simple "support" or "refute" labels fail to capture. This paper presents a framework for "faithful reasoning" that grounds LLMs in real-world literature to provide traceable, nuanced explanations.
Methodology: A Seven-Step Pipeline
The proposed system moves from a single claim to a comprehensive, evidence-backed verdict through a structured seven-step process:
- Claim Opposition: Generating the logical opposite of a claim to ensure a balanced evidence search.
- Retrieval: Querying academic databases for both the original and opposing claims.
- Takeaway Generation: Extracting the core message from retrieved abstracts relative to the claim.
- Local Labeling: Assigning a verdict on a 7-point scale with accompanying local explanations.
- Weighted Inference: Calculating a Reputation Score based on citation counts and journal impact factors to aggregate local labels into a "Global Label."
- Global Explanation: Synthesizing local explanations into a cohesive summary of scientific consensus.
- Attribution: Mapping every sentence in the final explanation back to its source abstract.
Experimental Setup
The framework was evaluated using 50 claims across five distinct domains: Climate, COVID-19, common Myths, Nutrients, and the SciFact benchmark. The evaluation involved 291 scientific articles, with an average of 5.8 articles retrieved per claim.
Results and Discussion
The system demonstrated significant improvements over standard LLM outputs in both accuracy and human trust.
Human Preference and Attribution
Human evaluators preferred the framework's "Global Explanations" over standard GPT-3.5 outputs 78% of the time, noting higher detail and instructiveness. Crucially, 80% of the sentences in these explanations were found to be correctly and faithfully attributed to source abstracts, effectively mitigating the risk of hallucination.
The Impact of Reputation Weighting
A key finding was the importance of source reputation. Reputation-weighted labels achieved an accuracy of 84%, a substantial increase from the 64% accuracy observed with unweighted labels. This confirms that incorporating metadata like impact factors is vital for determining scientific truth.
Discovering Nuance
The "anti-claim" strategy proved essential for uncovering scientific nuance. Over 51% of relevant documents were only discovered because the system explicitly searched for opposing evidence, allowing it to identify cases where a claim might be true in specific populations or conditions but not others.
Conclusion
By combining retrieval-augmented generation with reputation-based weighting and sentence-level attribution, this framework provides a path toward more reliable and transparent scientific fact-checking. Future work will focus on refining retrieval mechanisms and ensuring even stricter consistency between source takeaways and final syntheses.
Summary updated