An automated framework for assessing how well LLMs cite relevant medical references

Kevin Wu, Eric Wu, Kevin Wei, Angela Zhang, Allison Casasola, Teresa Nguyen, Sith Riantawan, Patricia Shi, Daniel Ho, James ZouNature Communications 2025 · OpenReview

Abstract

As large language models (LLMs) are increasingly used to address health-related queries, it is crucial that they support their conclusions with credible references. While models can cite sources, the extent to which these support claims remains unclear. To address this gap, we introduce SourceCheckup, an automated agent-based pipeline that evaluates the relevance and supportiveness of sources in LLM responses. We evaluate seven popular LLMs on a dataset of 800 questions and 58,000 pairs of statements and sources on data that represent common medical queries. Our findings reveal that between 50% and 90% of LLM responses are not fully supported, and sometimes contradicted, by the sources they cite. Even for GPT-4o with Web Search, approximately 30% of individual statements are unsupported, and nearly half of its responses are not fully supported. Independent assessments by doctors further validate these results. Our research underscores significant limitations in current LLMs to produce trustworthy medical references. Assessing the degree to which medical large language models reliably convey existing, trustworthy knowledge is crucial. This study introduces SourceCheckup, an automated framework revealing that large language models frequently cite medical references that do not fully support, or even contradict, their responses, showing significant gaps in reliability for clinical use.

Introduction

As Large Language Models (LLMs) like GPT-4o and Claude become common tools for healthcare information, the industry has often used "citations" as a proxy for trustworthiness. The prevailing assumption is that if a model provides a link to a reputable source, the information must be grounded in fact. However, LLMs are prone to hallucinations—generating plausible but false information. In the medical domain, this is a significant safety risk.

To address the prohibitive cost of manual verification by human experts, researchers developed SourceCheckup, an automated agent-based pipeline. Their findings serve as a wake-up call: between 50% and 90% of LLM-generated medical responses contain statements that are unsupported or even contradicted by their own cited sources.

The SourceCheckup Framework

SourceCheckup is a modular pipeline designed to mimic the rigorous fact-checking process of a medical professional. The workflow consists of four main stages:

  1. Question Generation: Creating realistic medical queries from both professional reference texts and real-world patient forums.
  2. LLM Answering: Posing these questions to various LLMs and extracting their responses along with cited URLs.
  3. Statement & Citation Parsing: Breaking down complex responses into individual, verifiable "atomic" statements and extracting the content from the cited web pages.
  4. Source Verification: Using a high-performing LLM (like GPT-4o or Claude 3.5) as an "evaluator agent" to determine if a specific source actually supports a specific statement.
The SourceCheckup pipeline: from question generation to automated source verification.

To ensure the automated evaluator was reliable, the team validated it against a panel of three US-licensed medical doctors. The automated system achieved an 88.7% agreement rate with the doctor consensus, which outperformed the agreement rate between individual doctors (approximately 86%).

Datasets and Evaluation

The study evaluated seven popular LLMs using a robust dataset comprising:

  • 800 Medical Questions: 400 generated from Mayo Clinic reference texts and 400 pulled from the "r/AskDocs" subreddit to represent messy, real-world patient queries.
  • 58,000 Statement-Source Pairs: A massive collection of individual claims matched against their cited evidence.
  • HealthSearchQA: A secondary dataset of consumer health questions used to further validate the findings.

Key Results

The quantitative results highlight a significant reliability gap in current AI models across three primary metrics: Citation URL Validity, Statement-level Support, and Response-level Support.

Evaluation of medical references across seven LLMs, showing scores for URL validity and support levels.
  • Response-Level Failure: Even for the best-performing model (GPT-4o with RAG), nearly half of its responses were not fully supported by the cited sources.
  • Statement-Level Support: For GPT-4o with Web Search, roughly 30% of individual statements were unsupported. For other models without active search, this failure rate was significantly higher.
  • URL Validity: Models without web access (API-only) produced valid URLs only 40% to 70% of the time, frequently hallucinating plausible-looking but non-existent links.

Discussion and Insights

  • RAG is not a Silver Bullet: Retrieval-Augmented Generation (RAG) drastically reduces "URL hallucinations" (linking to dead or fake sites). However, it does not guarantee that the model correctly interprets the found information. Models often find a high-quality source but then misstate its contents.
  • The "Reddit Gap": Models perform significantly worse on open-ended, user-generated questions from Reddit than on straightforward factual questions. This suggests LLMs struggle to synthesize evidence when faced with complex, multi-part clinical scenarios.
  • Decorative Citations: Many models provide a "laundry list" of citations at the end of a response. SourceCheckup reveals that many of these links are irrelevant to the specific claims made in the text, serving more as decorative elements than functional evidence.

Limitations and Future Work

The current framework primarily checks if one source supports one statement, rather than accounting for "aggregate support" from multiple sources. Additionally, over 90% of the cited sources were US-based, which may limit global applicability.

As a promising next step, the authors experimented with a SourceCleanup agent that uses evaluation feedback to automatically edit and correct unsupported statements. This agent achieved a 90.7% success rate in fixing errors, suggesting that automated auditing could eventually be used to self-correct medical AI in real-time.

Summary updated

Abstract

As large language models (LLMs) are increasingly used to address health-related queries, it is crucial that they support their conclusions with credible references. While models can cite sources, the extent to which these support claims remains unclear. To address this gap, we introduce SourceCheckup, an automated agent-based pipeline that evaluates the relevance and supportiveness of sources in LLM responses. We evaluate seven popular LLMs on a dataset of 800 questions and 58,000 pairs of statements and sources on data that represent common medical queries. Our findings reveal that between 50% and 90% of LLM responses are not fully supported, and sometimes contradicted, by the sources they cite. Even for GPT-4o with Web Search, approximately 30% of individual statements are unsupported, and nearly half of its responses are not fully supported. Independent assessments by doctors further validate these results. Our research underscores significant limitations in current LLMs to produce trustworthy medical references. Assessing the degree to which medical large language models reliably convey existing, trustworthy knowledge is crucial. This study introduces SourceCheckup, an automated framework revealing that large language models frequently cite medical references that do not fully support, or even contradict, their responses, showing significant gaps in reliability for clinical use.
BibTeX
@inproceedings{wu2025automated,
  author = {Kevin Wu and Eric Wu and Kevin Wei and Angela Zhang and Allison Casasola and Teresa Nguyen and Sith Riantawan and Patricia Shi and Daniel Ho and James Zou},
  title = {{An automated framework for assessing how well LLMs cite relevant medical references}},
  booktitle = {Nature Communications},
  year = {2025},
  url = {https://openreview.net/forum?id=LJC7wP5wGd},
}
Figure