Can ChatGPT explain it? Use of artificial intelligence in multiple sclerosis communication

Hernán Inojosa, Stephen Gilbert, Jakob Nikolas Kather, Undine Proschmann, Katja Akgün, Tjalf ZiemssenNeurological 2023 · DOI · OpenAlex

Introduction

Multiple Sclerosis (MS) is a complex neurological condition that frequently affects younger, tech-savvy populations. These patients often seek a deep understanding of their diagnosis and treatment options to participate actively in shared decision-making. However, the specialized language of neurology is notoriously difficult to translate into lay terms. While patients often turn to the internet for "second opinions," online medical resources are frequently written at a post-graduate level, far exceeding the average person's health literacy.

This study investigated whether Large Language Models (LLMs), specifically GPT-4, could serve as an effective "translator" for MS-related medical information, tailoring complex clinical scenarios for both healthcare professionals and patients.

Methodology

The researchers developed a structured evaluation framework centered on 64 common MS clinical scenarios. These scenarios spanned four critical domains: diagnosis, relapses, disease-modifying therapies (DMTs), and family planning.

Using a dual-output generation strategy, the team prompted GPT-4 to produce two distinct versions of text for each scenario: one intended for general practitioners (GPs) and another specifically targeted at a 6th-grade reading level for MS patients.

Example of GPT-4 generating dual-audience texts for a Clinically Isolated Syndrome (CIS) scenario.

The resulting 128 texts were then subjected to a rigorous independent review by three MS-specialized neurologists. Each text was scored on a 5-point Likert scale across three dimensions:

  1. Humanness: The naturalness of the style, tone, and expression.
  2. Correctness: The medical accuracy and clinical reliability of the information.
  3. Relevance: How well the content aligned with the intended subject and audience.

Additionally, five standardized readability metrics, including the Flesch-Kincaid Grade Level and the Gunning Fog Index, were used to objectively assess the linguistic complexity of the outputs.

Key Findings: The "Chameleon" Effect and the Accuracy Gap

The study revealed that GPT-4 is remarkably adept at adjusting its linguistic complexity—a phenomenon the authors describe as the "Chameleon" effect. The AI successfully shifted from a "16th-grade" (post-graduate) level for doctors to an "8th-to-9th-grade" level for patients. While it did not consistently hit the target 6th-grade level, the patient-facing text was significantly more accessible than standard medical documentation.

Quantitatively, the readability gap was stark. The Flesch-Kincaid Reading Ease score averaged ~15 (very difficult) for the professional texts compared to ~63 (standard/easy) for the patient-focused versions ($p < 0.001$).

However, this linguistic flexibility was decoupled from medical precision. While the AI achieved near-perfect scores for "Humanness" (median 5/5), its "Correctness" scores were lower and more variable (median 4.25/5). The researchers identified several "hallucinations"—substantive clinical errors presented with high confidence. These included incorrect dosing recommendations for specific DMTs (e.g., teriflunomide) and non-standard protocols for MRI monitoring using contrast agents.

Furthermore, the model struggled with regional regulatory nuances. It often failed to distinguish between FDA (US) and EMA (Europe) guidelines, presenting specific regional protocols as universal medical truths.

Conclusion and Future Directions

The study concludes that while LLMs like ChatGPT are powerful tools for drafting medical communications, they are not yet ready for autonomous clinical use. The "confident-but-wrong" nature of AI hallucinations poses a significant risk to patient safety if not mediated by human expertise.

Future improvements may lie in fine-tuning open-source models on curated, high-quality medical datasets from specialized centers to "ground" the AI in clinical reality. For now, ChatGPT should be viewed as a "supportive" technology—a sophisticated drafting assistant that must always remain tethered to expert neurological oversight.

Summary updated

BibTeX
@inproceedings{inojosa2023can,
  author = {Hernán Inojosa and Stephen Gilbert and Jakob Nikolas Kather and Undine Proschmann and Katja Akgün and Tjalf Ziemssen},
  title = {{Can ChatGPT explain it? Use of artificial intelligence in multiple sclerosis communication}},
  booktitle = {Neurological},
  year = {2023},
  doi = {10.1186/s42466-023-00270-8},
}
Figure