Are Generative AI systems Capable of Supporting Information Needs of Patients?

Shreya Rajagopal, Subhashis Hazarika, Sookyung Kim, Yan-Ming Chiou, Jae Ho Sohn, Hari Subramonyam, Shiwali Mohan

Introduction

In the contemporary healthcare landscape, patients frequently access their medical data—such as radiology reports and CT scans—through online portals long before they can consult with a physician. This delay creates a significant "information gap," often resulting in patient anxiety and a reliance on unverified online sources. While Large Language Models (LLMs) like GPT-4o and Claude 3.5 Sonnet demonstrate impressive performance on medical licensing exams, their ability to support a layperson's "sense-making" process remains largely untested. This research moves beyond academic benchmarks to perform a task-centric evaluation of Generative AI (GenAI) in supporting patient informational needs.

The complex clinical workflow and the resulting information gap for patients.

Methodology

The study employed a multi-stage approach to bridge the gap between AI capability and patient utility.

1. Formative Need-Finding

The researchers conducted simulated consultations where caregivers discussed a fictitious lung CT scan with a professional radiologist. Through inductive thematic analysis, they identified ten core informational needs, including clarifying medical terminology, understanding the implications of findings, and locating specific issues on the scan.

2. Comparative Evaluation and Dataset

Based on these needs, a high-quality multimodal dataset was developed. It consists of 25 visual question-answering (VQA) problems, each pairing a real-world patient question with two lung CT slices and a corresponding radiology report.

Example of the multimodal dataset: CT lung cancer screening report and corresponding axial images.

The performance of GPT-4o and Claude 3.5 Sonnet was evaluated in both "native" and "persona-prompted" modes. Responses were assessed for correctness by experts and analyzed for relevance and informativeness using Gricean maxims.

Key Findings: The Multimodality Gap

The evaluation revealed a stark contrast between the models' general medical knowledge and their practical utility in a clinical context.

The "Textbook Trap" and Verbosity

AI models frequently fell into a "textbook trap," providing generic information about diseases or statistics rather than tailoring answers to the specific patient's report. Furthermore, the models exhibited high verbosity; Claude 3.5 Sonnet, for instance, produced nearly double the word count of a human expert. While prompting for succinctness reduced length, it often led to the omission of clinically vital information.

Failure in Visual Grounding

Despite their multimodal architecture, both models struggled significantly with "Locating Issues." While a radiologist can intuitively point to "bubbly spots" or "white dots" on a scan, the AI models often provided generic instructions on how to read a scan or incorrectly claimed an inability to localize findings.

Error Rates and Metric Limitations

The study found alarmingly high error rates: 20% for GPT-4o and 40% for Claude 3.5 Sonnet. These errors included hallucinations and failures in causal reasoning. Crucially, the researchers noted that standard AI metrics, such as Universal Sentence Encoder (USE) similarity scores, failed to capture these clinical inaccuracies, often giving high scores to fundamentally incorrect answers.

Discussion: From Automation to Augmentation

The results suggest that current GenAI models are not yet ready for autonomous patient support. The authors argue for a philosophical shift from "AI-as-automation" (replacing the doctor) to "AI-as-augmentation." In this paradigm, AI is designed to prepare patients for their upcoming medical appointments, helping them formulate better questions and understand basic terminology without attempting to provide a definitive diagnosis or localize complex visual findings.

Conclusion

While GenAI holds promise for democratizing medical information, significant hurdles remain regarding accuracy and multimodal integration. Future work will focus on developing more robust measurement scales and exploring techniques like Retrieval-Augmented Generation (RAG) to mitigate the high error rates observed in "out-of-the-box" models.

Summary updated

BibTeX
@inproceedings{rajagopal2025are,
  author = {Shreya Rajagopal and Subhashis Hazarika and Sookyung Kim and Yan-Ming Chiou and Jae Ho Sohn and Hari Subramonyam and Shiwali Mohan},
  title = {{Are Generative AI systems Capable of Supporting Information Needs of Patients?}},
  booktitle = {IUI Workshops},
  year = {2025},
}
Figure