Retrieval-augmented systems can be dangerous medical communicators

Lionel Wong, Ayman Ali, Raymond Xiong, Shannon Zejiang Shen, Yoon Kim, Monica Agrawal

Abstract

Patients have long sought health information online, and increasingly, they are turning to generative AI to answer their health-related queries. Given the high stakes of the medical domain, techniques like retrieval-augmented generation and citation grounding have been widely promoted as methods to reduce hallucinations and improve the accuracy of AI-generated responses and have been widely adopted into search engines. This paper argues that even when these methods produce literally accurate content drawn from source documents sans hallucinations, they can still be highly misleading. Patients may derive significantly different interpretations from AI-generated outputs than they would from reading the original source material, let alone consulting a knowledgeable clinician. Through a large-scale query analysis on topics including disputed diagnoses and procedure safety, we support our argument with quantitative and qualitative evidence of the suboptimal answers resulting from current systems. In particular, we highlight how these models tend to decontextualize facts, omit critical relevant sources, and reinforce patient misconceptions or biases. We propose a series of recommendations -- such as the incorporation of communication pragmatics and enhanced comprehension of source documents -- that could help mitigate these issues and extend beyond the medical domain.

Introduction

This research, presented at ICML 2025, challenges the prevailing wisdom that Retrieval-Augmented Generation (RAG) is the "silver bullet" for AI safety. While RAG successfully reduces hallucinations by grounding responses in real-world documents, the authors argue that these systems are often "pragmatically misaligned"—producing factually accurate but dangerously misleading medical advice.

The core challenge is the gap between literal accuracy and communicative effectiveness. In the LLM world, "hallucination" is the primary villain. RAG was designed to fix this by forcing models to cite sources. However, a response can be 100% factually grounded yet still be a "dangerous communicator." For example, if a nervous patient asks about the "risks of surgery," a RAG system might provide a bulleted list of terrifying complications extracted from a reputable source. While "accurate," this response is pragmatically a failure if it omits the fact that those risks are rare and that the surgery is life-saving.

Figure 1: Different search paradigms lead to different outcomes. Vanilla search requires high health literacy, while current RAG can fuel anxiety by showing risks out of context. A pragmatic system would infer user intent and provide balanced context.

Methodology

The researchers conducted a large-scale query analysis of two major production RAG engines: Google’s AI Overview (AIO) and Perplexity AI. They used a "procedural query generation" approach to test three specific linguistic and pragmatic phenomena:

  1. Contextual Presupposition: Testing if the AI corrects a user who asks about symptoms of a "disputed" or pseudoscientific diagnosis (e.g., "adrenal fatigue").
  2. Query Valence/Bias: Asking "Why is [Procedure] safe?" vs. "Why is [Procedure] dangerous?" to see if the AI cherry-picks information to match the user's framing.
  3. Decontextualization: Analyzing whether the AI includes critical "balancing" information (like the rarity of a side effect or the benefits of a procedure) when listing complications.

The study utilized a benchmark of procedurally generated medical queries covering 13 disputed conditions and 28 medical procedures. To evaluate the thousands of responses, they employed an LLM-as-a-judge framework (using GPT-4o) to check for specific criteria, such as the mention of controversy, statistics, or counter-balancing benefits.

Key Insights and Results

The analysis revealed that RAG systems often fail to understand the intent behind a user's question and the intent of the source material.

Sycophancy and the Presupposition Trap

RAG systems exhibit a form of "confirmation bias" or sycophancy. If a user asks why a surgery is dangerous, the system doesn't just find a balanced article; it extracts only the "danger" sentences. The Jaccard similarity between sources cited for "safe" vs. "dangerous" queries was remarkably low (0.16 for Google AIO).

Furthermore, when a query presupposes a condition exists (e.g., "What are the symptoms of [Disputed Condition]?"), the AI often adopts that reality. It provides a list of symptoms as if they are settled science, failing to mention the condition is medically controversial. For disputed diagnoses, direct searches mentioned controversy 100% of the time, but this dropped significantly when the query presupposed the condition (56% for Google AIO and 69% for Perplexity).

Figure 2: Examples of RAG failures, including adopting pseudoscientific diagnoses like 'rope worms', failing to disambiguate vague terms for millennials, and providing biased responses for mastectomy safety.

Missing Context and Statistics

Systems often fail to provide the statistical context necessary for risk assessment. In responses about surgical complications, statistics regarding rarity were mentioned in only 4% (Google) and 5% (Perplexity) of cases. Even when the underlying source document emphasized benefits over risks, the RAG-generated summaries mentioned benefits in only 6-10% of responses to risk-focused queries.

Discussion and Future Work

Current RAG systems lack "pragmatic reasoning"—they do not use "Rational Speech Acts" to infer why a patient is asking and provide a response that addresses the underlying concern. The authors suggest that future systems should be designed to reason about communicative intent. This involves using frameworks like Bayesian models of pragmatics to "read between the lines" of a user's query.

They also recommend UI changes that situate extracted facts within the broader context of the source document, rather than presenting them as isolated, authoritative bullets. This would help mitigate the "common" fallacy, where vague terms lead to a skewed perception of risk.

This paper was presented at ICML 2025.

Summary updated

Abstract

Patients have long sought health information online, and increasingly, they are turning to generative AI to answer their health-related queries. Given the high stakes of the medical domain, techniques like retrieval-augmented generation and citation grounding have been widely promoted as methods to reduce hallucinations and improve the accuracy of AI-generated responses and have been widely adopted into search engines. This paper argues that even when these methods produce literally accurate content drawn from source documents sans hallucinations, they can still be highly misleading. Patients may derive significantly different interpretations from AI-generated outputs than they would from reading the original source material, let alone consulting a knowledgeable clinician. Through a large-scale query analysis on topics including disputed diagnoses and procedure safety, we support our argument with quantitative and qualitative evidence of the suboptimal answers resulting from current systems. In particular, we highlight how these models tend to decontextualize facts, omit critical relevant sources, and reinforce patient misconceptions or biases. We propose a series of recommendations -- such as the incorporation of communication pragmatics and enhanced comprehension of source documents -- that could help mitigate these issues and extend beyond the medical domain.
BibTeX
@misc{wong2025retrieval,
  author = {Lionel Wong and Ayman Ali and Raymond Xiong and Shannon Zejiang Shen and Yoon Kim and Monica Agrawal},
  title = {{Retrieval-augmented systems can be dangerous medical communicators}},
  year = {2025},
  eprint = {2502.14898},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.14898},
}
Figure