Faithful Reasoning over Scientific Claims

Neset Özkan Tan, Niket Tandon, David Wadden, Oyvind Tafjord, Mark Gahegan, Michael Witbrock

Introduction

Verifying scientific claims is an increasingly complex challenge due to the exponential growth of research literature. While Large Language Models (LLMs) offer potential assistance, their application in scientific contexts is hindered by hallucinations and a lack of traceability. Furthermore, scientific truth is often non-binary, requiring a nuanced understanding of underlying assumptions that simple "support" or "refute" labels fail to capture. This paper presents a framework for "faithful reasoning" that grounds LLMs in real-world literature to provide traceable, nuanced explanations.

Methodology: A Seven-Step Pipeline

The proposed system moves from a single claim to a comprehensive, evidence-backed verdict through a structured seven-step process:

  1. Claim Opposition: Generating the logical opposite of a claim to ensure a balanced evidence search.
  2. Retrieval: Querying academic databases for both the original and opposing claims.
  3. Takeaway Generation: Extracting the core message from retrieved abstracts relative to the claim.
  4. Local Labeling: Assigning a verdict on a 7-point scale with accompanying local explanations.
  5. Weighted Inference: Calculating a Reputation Score based on citation counts and journal impact factors to aggregate local labels into a "Global Label."
  6. Global Explanation: Synthesizing local explanations into a cohesive summary of scientific consensus.
  7. Attribution: Mapping every sentence in the final explanation back to its source abstract.
The framework architecture for faithful reasoning, illustrating the flow from claim to global explanation.

Experimental Setup

The framework was evaluated using 50 claims across five distinct domains: Climate, COVID-19, common Myths, Nutrients, and the SciFact benchmark. The evaluation involved 291 scientific articles, with an average of 5.8 articles retrieved per claim.

Results and Discussion

The system demonstrated significant improvements over standard LLM outputs in both accuracy and human trust.

Human Preference and Attribution

Human evaluators preferred the framework's "Global Explanations" over standard GPT-3.5 outputs 78% of the time, noting higher detail and instructiveness. Crucially, 80% of the sentences in these explanations were found to be correctly and faithfully attributed to source abstracts, effectively mitigating the risk of hallucination.

The Impact of Reputation Weighting

A key finding was the importance of source reputation. Reputation-weighted labels achieved an accuracy of 84%, a substantial increase from the 64% accuracy observed with unweighted labels. This confirms that incorporating metadata like impact factors is vital for determining scientific truth.

Performance comparison across diverse scientific datasets, showing high human preference for the framework's explanations and attributions.

Discovering Nuance

The "anti-claim" strategy proved essential for uncovering scientific nuance. Over 51% of relevant documents were only discovered because the system explicitly searched for opposing evidence, allowing it to identify cases where a claim might be true in specific populations or conditions but not others.

Conclusion

By combining retrieval-augmented generation with reputation-based weighting and sentence-level attribution, this framework provides a path toward more reliable and transparent scientific fact-checking. Future work will focus on refining retrieval mechanisms and ensuring even stricter consistency between source takeaways and final syntheses.

Summary updated

Abstract

Claim verification in scientific domains requires models that faithfully incorporate relevant knowledge from the ever-growing, vast existing literature. Unfaithful claim verifications can lead to misinformation such as those observed during the COVID-19 pandemic. Fact-checking systems often fail to capture the complex relationship between claims and evidence, especially with ambiguous claims and implicit assumptions. Relying only on current LLMs poses challenges due to hallucinations and information traceability issues. To address these challenges, our approach considers multiple viewpoints onto the scientific literature, enabling the assessment of contradictory arguments and implicit assumptions. Our proposed inference method adds faithful reasoning to large language models by distilling information from diverse, relevant scientific abstracts. This method provides a verdict label that can be weighted by the reputation of the scientific articles and an explanation that can be traced back to sources. Our findings demonstrate that humans not only perceive our explanation to be significantly superior to the off-the-shelf model, but they also evaluate it as faithfully enabling the tracing of evidence back to its original sources.
BibTeX
@inproceedings{tan2024faithful,
  author = {Neset Özkan Tan and Niket Tandon and David Wadden and Oyvind Tafjord and Mark Gahegan and Michael Witbrock},
  title = {{Faithful Reasoning over Scientific Claims}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaaiss.v3i1.31209},
}
Figure