SciClaims: An End-to-End Generative System for Biomedical Claim Analysis

Raúl Ortega, Jose Manuel Gomez-PerezEMNLP 2025

TL;DR

This paper introduces SciClaims, an end-to-end system that automates the analysis of scientific claims in biomedical texts. By leveraging a single large language model, SciClaims extracts claims, retrieves supporting evidence from a biomedical corpus, and verifies their veracity, offering a user-friendly interface for real-time analysis. This system aims to streamline laborious tasks like systematic literature reviews and patent validation, making scientific knowledge discovery more efficient and trustworthy.

What Problem Does This Paper Address?

The paper addresses the significant challenge of validating scientific claims within the ever-growing volume of biomedical literature. Tasks like Systematic Literature Reviews (SLRs) and patent validation are crucial for clinical decisions, regulatory submissions, and research and development. However, manually verifying claims is labor-intensive, prone to errors, and difficult to scale. Existing approaches often tackle claim extraction, evidence retrieval, and verification as separate, multi-stage pipelines, which can be complex and prone to failure. The authors aim to create a seamless, end-to-end system that integrates these steps efficiently.

Methods

SciClaims employs a unified approach using a single large language model (LLM), specifically Llama3 8B Instruct, to handle the entire claim analysis pipeline. The system architecture comprises three main modules:

  1. Claim Extraction: This module identifies potential factual claims within the input text. It uses the LLM in two steps: an initial generation of claims and a subsequent refinement to improve quality. The prompts for this module were carefully tuned based on evaluations.
  2. Document Retrieval: Once claims are extracted, this module searches a curated biomedical corpus (4.7 million PubMed abstracts from 2000-2022) for relevant evidence. It utilizes an Elasticsearch-based retrieval engine, querying with the extracted claim. The dataset is filtered for highly influential citations to ensure quality.
  3. Claim Verification: The final module uses the LLM again to verify the extracted claim against the retrieved evidence. It assigns one of three labels: SUPPORT, REFUTE, or NEI (Not Enough Information). Crucially, the system also generates a natural language rationale explaining its decision and highlights the most influential sentences from the evidence.

The system is optimized to run efficiently on a single GPU and is accessible via a web-based interface.

Datasets

  • PubMed Abstracts (2000-2022): Used as the corpus for evidence retrieval. This dataset was curated using Semantic Scholar's Highly Influential Citations metric.
  • SciFact: A benchmark dataset for evaluating claim verification. It contains 1,400 expert-authored scientific claims linked to evidence-containing abstracts and verification labels.

Key Insights and Ideas

  • Unified LLM Approach: The core innovation is using a single LLM for the entire pipeline (extraction, retrieval, verification), eliminating the need for complex multi-stage systems and additional fine-tuning for each component. This simplifies the architecture and potentially improves end-to-end coherence.
  • Focus on Explainability: SciClaims doesn't just provide a label; it offers natural language rationales and highlights supporting evidence, enhancing transparency and trustworthiness.
  • Efficiency on a Single GPU: The system is optimized for practical deployment, capable of running efficiently on a single GPU, making it accessible for real-world applications.
  • Claim Definition Properties (CDP) and Claim Refinement (CR): These prompt engineering techniques were developed to guide the LLM in generating higher-quality, more precise, and verifiable claims, significantly improving performance.

Key Results

  • Claim Extraction: SciClaims achieved superior scores in ROUGE-1, ROUGE-L, and semantic similarity compared to baselines when evaluated against SciFact, indicating more faithful and informative claim generation. The CDP and CR refinements led to a substantial increase in correctly generated claims (76.36% with CDP+CR).
  • Claim Verification: SciClaims demonstrated competitive performance against fine-tuned models on the SciFact dataset, even in a zero-shot setting, highlighting its generalization ability. It also produced significantly fewer "Not Enough Information" (NEI) labels than a strong baseline (MultiVerS), providing more definitive results.
  • End-to-End Performance: The system achieved a high overall system score, balancing accuracy and efficiency. The SciClaims+CDP+CR configuration, when used with SciClaims for both extraction and verification, was the only one capable of running on a single 24GB GPU, making it uniquely suitable for real-world deployment.
  • Human Evaluation: Human annotators showed high alignment with the LLM judge's evaluations, rating the accuracy of claims and rationales highly.

Limitations and Future Work

  • Document Retrieval: While SciClaims performs well, some QA-oriented methods generated more compact, less specific claims that might increase the likelihood of retrieving related documents.
  • Rationale Completeness: Human annotators noted that while rationales were relevant and justified, their completeness could be improved.
  • Future Work: The authors suggest further exploration of prompt engineering techniques and the potential for integrating more sophisticated retrieval methods. The system is presented as a robust tool ready for real-world applications in pharmaceutical patent analysis and systematic literature reviews, with ongoing exploration in these areas.

Summary updated

BibTeX
@inproceedings{ortega2025sciclaims,
  author = {Raúl Ortega and Jose Manuel Gomez-Perez},
  title = {{SciClaims: An End-to-End Generative System for Biomedical Claim Analysis}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure