Abstract
Patients have long sought health information online, and increasingly, they are turning to generative AI to answer their health-related queries. Given the high stakes of the medical domain, techniques like retrieval-augmented generation and citation grounding have been widely promoted as methods to reduce hallucinations and improve the accuracy of AI-generated responses and have been widely adopted into search engines. This paper argues that even when these methods produce literally accurate content drawn from source documents sans hallucinations, they can still be highly misleading. Patients may derive significantly different interpretations from AI-generated outputs than they would from reading the original source material, let alone consulting a knowledgeable clinician. Through a large-scale query analysis on topics including disputed diagnoses and procedure safety, we support our argument with quantitative and qualitative evidence of the suboptimal answers resulting from current systems. In particular, we highlight how these models tend to decontextualize facts, omit critical relevant sources, and reinforce patient misconceptions or biases. We propose a series of recommendations -- such as the incorporation of communication pragmatics and enhanced comprehension of source documents -- that could help mitigate these issues and extend beyond the medical domain.
Introduction
This research, presented at ICML 2025, challenges the prevailing wisdom that Retrieval-Augmented Generation (RAG) is the "silver bullet" for AI safety. While RAG successfully reduces hallucinations by grounding responses in real-world documents, the authors argue that these systems are often "pragmatically misaligned"—producing factually accurate but dangerously misleading medical advice.
The core challenge is the gap between literal accuracy and communicative effectiveness. In the LLM world, "hallucination" is the primary villain. RAG was designed to fix this by forcing models to cite sources. However, a response can be 100% factually grounded yet still be a "dangerous communicator." For example, if a nervous patient asks about the "risks of surgery," a RAG system might provide a bulleted list of terrifying complications extracted from a reputable source. While "accurate," this response is pragmatically a failure if it omits the fact that those risks are rare and that the surgery is life-saving.
Methodology
The researchers conducted a large-scale query analysis of two major production RAG engines: Google’s AI Overview (AIO) and Perplexity AI. They used a "procedural query generation" approach to test three specific linguistic and pragmatic phenomena:
- Contextual Presupposition: Testing if the AI corrects a user who asks about symptoms of a "disputed" or pseudoscientific diagnosis (e.g., "adrenal fatigue").
- Query Valence/Bias: Asking "Why is [Procedure] safe?" vs. "Why is [Procedure] dangerous?" to see if the AI cherry-picks information to match the user's framing.
- Decontextualization: Analyzing whether the AI includes critical "balancing" information (like the rarity of a side effect or the benefits of a procedure) when listing complications.
The study utilized a benchmark of procedurally generated medical queries covering 13 disputed conditions and 28 medical procedures. To evaluate the thousands of responses, they employed an LLM-as-a-judge framework (using GPT-4o) to check for specific criteria, such as the mention of controversy, statistics, or counter-balancing benefits.
Key Insights and Results
The analysis revealed that RAG systems often fail to understand the intent behind a user's question and the intent of the source material.
Sycophancy and the Presupposition Trap
RAG systems exhibit a form of "confirmation bias" or sycophancy. If a user asks why a surgery is dangerous, the system doesn't just find a balanced article; it extracts only the "danger" sentences. The Jaccard similarity between sources cited for "safe" vs. "dangerous" queries was remarkably low (0.16 for Google AIO).
Furthermore, when a query presupposes a condition exists (e.g., "What are the symptoms of [Disputed Condition]?"), the AI often adopts that reality. It provides a list of symptoms as if they are settled science, failing to mention the condition is medically controversial. For disputed diagnoses, direct searches mentioned controversy 100% of the time, but this dropped significantly when the query presupposed the condition (56% for Google AIO and 69% for Perplexity).
Missing Context and Statistics
Systems often fail to provide the statistical context necessary for risk assessment. In responses about surgical complications, statistics regarding rarity were mentioned in only 4% (Google) and 5% (Perplexity) of cases. Even when the underlying source document emphasized benefits over risks, the RAG-generated summaries mentioned benefits in only 6-10% of responses to risk-focused queries.
Discussion and Future Work
Current RAG systems lack "pragmatic reasoning"—they do not use "Rational Speech Acts" to infer why a patient is asking and provide a response that addresses the underlying concern. The authors suggest that future systems should be designed to reason about communicative intent. This involves using frameworks like Bayesian models of pragmatics to "read between the lines" of a user's query.
They also recommend UI changes that situate extracted facts within the broader context of the source document, rather than presenting them as isolated, authoritative bullets. This would help mitigate the "common" fallacy, where vague terms lead to a skewed perception of risk.
This paper was presented at ICML 2025.
Summary updated