Self-Diagnosis and Large Language Models: A New Front for Medical Misinformation

Francois Barnard, Marlize Van Sittert, Sirisha RambhatlaarXiv 2023 · arXiv · OpenReview

Abstract

Improving healthcare quality and access remains a critical concern for countries worldwide. Consequently, the rise of large language models (LLMs) has erupted a wealth of discussion around healthcare applications among researchers and consumers alike. While the ability of these models to pass medical exams has been used to argue in favour of their use in medical training and diagnosis, the impact of their inevitable use as a self-diagnostic tool and their role in spreading healthcare misinformation has not been evaluated. In this work, we critically evaluate LLMs' capabilities from the lens of a general user self-diagnosing, as well as the means through which LLMs may aid in the spread of medical misinformation. To accomplish this, we develop a testing methodology which can be used to evaluate responses to open-ended questions mimicking real-world use cases. In doing so, we reveal that a) these models perform worse than previously known, and b) they exhibit peculiar behaviours, including overconfidence when stating incorrect recommendations, which increases the risk of spreading medical misinformation.

Introduction

The rapid advancement of Large Language Models (LLMs) has sparked significant interest in their potential healthcare applications. While models like GPT-4 have demonstrated the ability to pass rigorous medical licensing exams, such as the USMLE, these successes may provide a false sense of security regarding their clinical utility. Standardized exams rely on structured, multiple-choice formats that do not reflect the messy, narrative nature of patient-led self-diagnosis. This research introduces EvalPrompt, a framework designed to evaluate LLMs in the context of open-ended medical queries, revealing critical vulnerabilities in clarity, correctness, and robustness.

The EvalPrompt Methodology

To bridge the gap between exam performance and real-world application, the researchers developed EvalPrompt, a two-stage evaluation pipeline.

The EvalPrompt methodology, encompassing open-ended prompt generation (Experiment 1) and sentence dropout analysis (Experiment 2).

The first stage involves transforming structured USMLE Step 1 questions into open-ended prompts. By removing multiple-choice options and rephrasing questions to require narrative synthesis, the framework tests the model's ability to generate accurate medical advice without predefined cues. The second stage utilizes "sentence dropout" to simulate information scarcity—a common occurrence when patients describe symptoms—to assess the model's robustness.

Human-in-the-Loop Assessment

A critical component of EvalPrompt is the involvement of human assessors. Six individuals—three medical experts and three non-experts—evaluated the model's responses. Answers were categorized into four groups: Correct (C), Partially Correct (PC), Incorrect (I), and Ambiguous (A). This multi-perspective approach highlights the "clarity gap" between what a layperson might perceive as a helpful answer and what a professional identifies as clinically sound.

Key Findings: The Reality of Open-Ended Performance

The results indicate that LLM performance drops significantly when the "crutch" of multiple-choice options is removed.

Distribution of answer assessments across three medical experts and three non-experts for open-ended USMLE prompts.

While GPT-4 is often cited for its high USMLE scores, this study found that it achieved a unanimous "Correct" rating from all six assessors for only 31% (29 out of 94) of the questions. Furthermore, there was a stark disconnect in perception: agreement between experts and non-experts on response correctness was only 34%. Non-experts were more likely to trust vague or ambiguous responses that experts flagged as potentially dangerous or inaccurate.

Robustness and the Risk of Overconfidence

The ablation study (Experiment 2) revealed a paradoxical finding regarding model robustness. GPT-4 remained "correct" in its diagnosis 61% of the time even when critical clinical details were removed from the prompt. While this suggests a high degree of internal knowledge, it also points to a dangerous level of overconfidence. In a self-diagnosis scenario, a model that provides a definitive recommendation despite lacking a full clinical picture significantly increases the risk of spreading medical misinformation.

Conclusion and Future Directions

The EvalPrompt framework demonstrates that current LLMs are not yet reliable enough for autonomous medical self-diagnosis. The discrepancy between exam scores and open-ended accuracy suggests that "passing" a test is an insufficient metric for clinical safety. Future work must focus on developing datasets that mirror the ambiguity of real-world patient interactions and training models to prioritize asking clarifying questions over providing potentially misinformed diagnoses.

Summary updated

Abstract

Improving healthcare quality and access remains a critical concern for countries worldwide. Consequently, the rise of large language models (LLMs) has erupted a wealth of discussion around healthcare applications among researchers and consumers alike. While the ability of these models to pass medical exams has been used to argue in favour of their use in medical training and diagnosis, the impact of their inevitable use as a self-diagnostic tool and their role in spreading healthcare misinformation has not been evaluated. In this work, we critically evaluate LLMs' capabilities from the lens of a general user self-diagnosing, as well as the means through which LLMs may aid in the spread of medical misinformation. To accomplish this, we develop a testing methodology which can be used to evaluate responses to open-ended questions mimicking real-world use cases. In doing so, we reveal that a) these models perform worse than previously known, and b) they exhibit peculiar behaviours, including overconfidence when stating incorrect recommendations, which increases the risk of spreading medical misinformation.
BibTeX
@misc{barnard2023self,
  author = {Francois Barnard and Marlize Van Sittert and Sirisha Rambhatla},
  title = {{Self-Diagnosis and Large Language Models: A New Front for Medical Misinformation}},
  year = {2023},
  eprint = {2307.04910},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.04910},
}
Figure