Fact Check: Assessing the Response of ChatGPT to Alzheimer’s Disease Statements with Varying Degrees of Misinformation

Sean S. Huang, Qingyuan Song, Kimberly J. Beiting, Maria Duggan, Kristin Hines, Harvey J. Murff, Vania Leung, James S. Powers, T.s. Harvey, Bradley Malin, Zhijun YinmedRxiv 2023 · DOI

Abstract

The majority of surveyed healthcare professionals acknowledged the potential value of ChatGPT in mitigating Alzheimer’s Disease (AD) misinformation. However, the need for more refined and detailed explanations of the disease's mechanisms and treatments was highlighted.

Introduction

As health information becomes increasingly democratized online, the risk of misinformation grows proportionally. With over 80% of American adults seeking health advice on the internet, many are now turning to Large Language Models (LLMs) like ChatGPT as a primary resource. For complex and stigmatized conditions such as Alzheimer’s Disease (AD), the stakes are exceptionally high: misinformation can lead to delayed diagnoses, caregiver burnout, or the pursuit of unproven and potentially dangerous "cures."

This study evaluates GPT-4’s ability to act as a reliable "fact-checker" for AD. The researchers sought to determine if the model could not only identify common myths but also explain them accurately and translate medical jargon into language accessible to a layperson—specifically at a middle-school reading level—without losing essential clinical information.

Methodology

The researchers conducted a cross-sectional study using a "human-in-the-loop" evaluation framework. The process involved four key stages:

  1. Myth Selection: A curated list of 20 common AD myths was developed, covering topics from genetics and diagnosis to lifestyle and treatment (e.g., "AD is a normal part of aging," "Supplements can cure AD").
  2. Prompting: GPT-4 was prompted to provide an opinion on each myth and then asked to rewrite that explanation for a middle-school education level.
  3. Expert Review: 11 clinicians, including attending geriatricians and medical trainees, reviewed the outputs using a REDCap-based survey.
  4. Scoring: Clinicians used a 5-point Likert scale (-2 to 2) to evaluate the accuracy, readability, and information retention of the simplified text.

Key Results and Clinical Insights

Accuracy and Identification

GPT-4 demonstrated a high level of basic accuracy, correctly identifying 100% (20/20) of the myths as inaccurate. The mean satisfaction score for the model's explanations was 1.0 (±0.3) on a scale of -2 to 2, indicating general approval from the clinical panel.

The "Technically Correct" Trap

Despite the high identification rate, clinicians disagreed with the model's specific reasoning on 7 out of the 20 myths. The most significant point of contention involved Myth #7 ("AD is not fatal"). GPT-4 argued that AD is not a direct cause of death but rather contributes to complications like pneumonia. Clinicians gave this response the lowest score (0.2), noting that this distinction is a "harmful fallacy." In a clinical context, such an explanation might lead caregivers to believe death is avoidable if they simply prevent secondary complications, ignoring the terminal nature of the disease itself.

Empathy and Stigma

Interestingly, the model performed exceptionally well in navigating the social dimensions of the disease. For Myth #5 ("AD is a shameful diagnosis"), GPT-4 received a high score of 1.6. It successfully mirrored human-like empathy and emphasized compassion, suggesting that LLMs may be effective at addressing the stigma associated with neurodegenerative conditions.

Simplification vs. Nuance

The model was highly effective at simplifying language, with 81% of clinicians finding the simplified text highly readable and 82% agreeing that it retained the original message. However, clinicians noted that GPT-4 occasionally struggled with the "spectrum" of aging. It sometimes failed to clearly distinguish between normal age-related cognitive decline and the early stages of AD, a nuance that is vital for preventing either unnecessary alarm or a false sense of security.

Conclusion

The study had a notable impact on the participating clinicians' perceptions of AI. Before the survey, many were skeptical; however, after reviewing the outputs, 82% agreed or strongly agreed that ChatGPT has the potential to mitigate AD misinformation.

While GPT-4 proves to be a powerful tool for public health education, the findings underscore that "correct" identification does not equal a "perfect" explanation. The authors conclude that LLMs still require domain expert verification to ensure that the critical nuances of life-and-death medical conditions are preserved during the simplification process. Future work should focus on refining the model's ability to explain complex disease mechanisms and treatments.

Summary updated

Abstract

The majority of surveyed healthcare professionals acknowledged the potential value of ChatGPT in mitigating AD misinformation. However, the need for more refined and detailed explanations of the disease's mechanisms and treatments was highlighted.
BibTeX
@misc{huang2023fact,
  author = {Sean S. Huang and Qingyuan Song and Kimberly J. Beiting and Maria Duggan and Kristin Hines and Harvey J. Murff and Vania Leung and James S. Powers and T.S. Harvey and Bradley Malin and Zhijun Yin},
  title = {{Fact Check: Assessing the Response of ChatGPT to Alzheimer’s Disease Statements with Varying Degrees of Misinformation}},
  year = {2023},
  doi = {10.1101/2023.09.04.23294917},
  howpublished = {medRxiv},
}
Figure