Large language models management of complex medication regimens: a case-based evaluation

Steven Xu, Amoreena Most, Aaron Chase, Tanner Hedrick, Brian Murray, Kelli Keats, Susan Smith, Erin Barreto, Tianming Liu, Andrea SikoramedRxiv 2024 · DOI

TL;DR

While Large Language Models (LLMs) excel at medical exams, this study reveals they struggle with the high-stakes reality of ICU medication management. Even the top performer, GPT-4, suggested life-threatening errors in over 12% of cases, suggesting that while AI has potential as a clinical assistant, it is currently too unreliable for autonomous pharmacy tasks.

What Problem Does This Paper Address?

The medical community has been abuzz with LLMs passing board exams and diagnosing rare diseases. However, there is a massive gap between "knowing" medicine and "practicing" it—specifically in Comprehensive Medication Management (CMM).

CMM is the complex process of ensuring every drug a patient takes is appropriate, effective, and safe given their specific comorbidities and other medications. This is exceptionally difficult in the Intensive Care Unit (ICU), where patients are unstable, organ systems are failing, and the "medication list" can be dozens of lines long. The authors sought to determine if current state-of-the-art LLMs can identify medication errors and optimize regimens for these "final boss" clinical scenarios.

Methods

The researchers used a case-based evaluation strategy:

  1. Case Development: Critical care pharmacists created eight detailed ICU patient cases, including history of present illness, vitals, lab values, and current medications.
  2. Prompting Strategy: They employed a "two-step" prompting process. First, they used single-shot training, providing the LLM with one example case and its "ground truth" (the correct clinical plan). Second, they used Chain-of-Thought (CoT) prompting, asking the models to show their reasoning as they developed medication plans for the remaining seven test cases.
  3. Expert Review: A panel of seven board-certified critical care pharmacists reviewed the AI-generated plans. They rated whether they would "continue" or "discontinue" each recommendation and flagged errors.
  4. Models Tested: ChatGPT (GPT-3.5), ChatGPT (GPT-4), Claude 2, and Llama 2-70b.

Datasets

The study did not use a standard public dataset. Instead, it introduced a bespoke benchmark of eight complex ICU cases. These cases were designed to be clinically rigorous, featuring challenges like sepsis, intracranial pressure management, and acute kidney injury.

Key Insights and Ideas

  • The "Dose Makes the Poison": The most chilling insight was the nature of the errors. In one instance, an LLM recommended a 250 mL bolus of 23% hypertonic saline for a patient with brain swelling. In clinical practice, the standard dose is 30 mL; a 250 mL dose would likely be fatal.
  • GPT-4 is the "Valedictorian," but still fails: GPT-4 significantly outperformed the other models in terms of clinical relevance and safety. However, "better than the others" is not the same as "safe for patients."
  • Reasoning vs. Hallucination: The use of Chain-of-Thought reasoning helped models structure their output, but it didn't prevent them from confidently asserting incorrect—and dangerous—pharmacological "facts."
  • The Agreement Gap: Clinicians only agreed with the AI's overall regimens at a very low level (averaging around 2 on a 5-point Likert scale). This suggests a fundamental disconnect between how LLMs prioritize therapy and how human experts manage critical illness.

Key Results

The quantitative performance of the models was sobering:

  • Rate of Continuation: Pharmacists only agreed to continue therapies recommended by the LLMs 55.8% to 67.9% of the time.
  • Life-Threatening Errors: These were present in 12.2% of GPT-4's recommendations, jumping to a staggering 46.9% for Claude 2 and 38.8% for GPT-3.5.
  • Medication Errors: Clinicians perceived between 1.57 and 4.29 errors per regimen across the different models.
  • Statistical Significance: GPT-4 had significantly fewer life-threatening errors than GPT-3.5 or Claude 2 (p=0.0007), marking it as the only model currently showing even a glimmer of potential for clinical decision support.

Limitations and Future Work

The authors acknowledge several key limitations:

  • Small Sample Size: With only eight cases and seven reviewers, the study is a "pilot" evaluation rather than a definitive trial.
  • Information Silos: The LLMs were given "snapshots" of data. In a real ICU, pharmacists look at trends over time (e.g., is the creatinine rising or falling?) which were not fully captured here.
  • Lack of Fine-Tuning: The models used were general-purpose. The authors suggest that domain-specific training on high-quality pharmaceutical datasets might drastically improve safety.

Conclusion: The paper concludes that while LLMs might one day serve as a "second pair of eyes" for pharmacists, they currently lack the precision required for ICU care. For now, the "human in the loop" isn't just a preference—it's a safety requirement.

Summary updated

Abstract

Abstract Background Large language models (LLMs) have shown capability in diagnosing complex medical cases and passing medical licensing exams, but to date, only limited evaluations have studied how LLMs interpret, analyze, and optimize complex medication regimens. The purpose of this evaluation was to test four LLMs ability to identify medication errors and appropriate medication interventions on complex patient cases from the intensive care unit (ICU). Methods A series of eight patient cases were developed by critical care pharmacists including history of present illness, laboratory values, vital signs, and medication regimens. Then, four LLMs (ChatGPT (GPT-3.5), ChatGPT (GPT-4), Claude2, and Llama2-7b) were prompted to develop a medication regimen for the patient. LLM generated medication regimens were then reviewed by a panel of seven critical care pharmacists to assess for presence of medication errors and clinical relevance. For each medication regimen recommended by the LLM, clinicians were asked to assess for if they would continue a medication, identify perceived medication errors in the medications recommended, identify the presence of life-threatening medication choices, and rank overall agreement on a 5-point Likert scale. Results The clinician panel rated to continue therapies recommended by the LLMs between 55.8-67.9% of the time. Clinicians perceived between 1.57-4.29 medication errors per recommended regimen, and life-threatening recommendations were present between 15.0-55.3% of the time. Level agreement was between 1.85-2.67 for the four LLMs. Conclusions LLMs demonstrated potential to serve as clinical decision support for the management of complex medication regimens with further domain specific training; however, caution should be used when employing LLMs for medication management given the present capabilities.
BibTeX
@misc{xu2024large,
  author = {Steven Xu and Amoreena Most and Aaron Chase and Tanner Hedrick and Brian Murray and Kelli Keats and Susan Smith and Erin Barreto and Tianming Liu and Andrea Sikora},
  title = {{Large language models management of complex medication regimens: a case-based evaluation}},
  year = {2024},
  doi = {10.1101/2024.07.03.24309889},
  howpublished = {medRxiv},
}
Figure