Scalable medication extraction and discontinuation identification from electronic health records using large language models

Chong Shao, Douglas Snyder, Chao Li, Bowen Gu, Kerry Ngan, Chun‐Ting Yang, Jiageng Wu, Richard Wyss, Kueiyu Joshua Lin, Jie Yang

TL;DR

This paper benchmarks 12 state-of-the-art Large Language Models (LLMs) on their ability to extract medications and identify discontinuations from unstructured clinical notes. The researchers found that while GPT-4o leads the pack, top-tier open-source models like Llama-3.1-70B and Qwen2.5-72B are now highly competitive, offering a scalable, privacy-conscious path for automating drug-safety surveillance and clinical decision support.

What Problem Does This Paper Address?

Electronic Health Records (EHRs) are filled with critical patient data, but much of it is "buried" in unstructured clinical notes. Specifically, identifying when a patient stops taking a medication is vital for preventing adverse drug events and understanding treatment failures.

Standard structured data (like insurance claims) often misses these nuances—claims show when a prescription is filled, but not necessarily when a doctor tells a patient to stop taking it due to a side effect. Manually extracting this information is too slow and expensive to scale. This paper investigates whether LLMs can bridge this gap by accurately identifying medication names and their status (active, discontinued, or neither) without requiring massive human-annotated training sets.

Methods

The authors conducted a systematic benchmark of 12 LLMs across three primary tasks:

  1. Medication Extraction: Identifying drug names in text (Named Entity Recognition).
  2. Isolated Status Classification: Given a drug name, classifying it as "active," "discontinued," or "neither" based on context.
  3. Joint Task (End-to-End): Extracting the drug and classifying its status in one go—the most realistic "real-world" scenario.

They tested three prompting strategies: Zero-shot (no examples), 5-shot (five examples provided), and Chain-of-Thought (CoT) (asking the model to "think step-by-step"). The model lineup included proprietary giants (GPT-4o), general-purpose open-source models (Llama 3.1, Qwen 2.5, Mistral), and medical-specific models (Me-LLaMA, Meditron).

Datasets

The study utilized three distinct datasets to ensure robustness:

  • Re-CASI: A publicly available, de-identified dataset derived from the Clinical Acronym Sense Inventory.
  • MIV-Med (New): A newly annotated dataset created by the authors using discharge summaries from the public MIMIC-IV database. It was annotated by experts, including an MD.
  • MGB-Med (New): An internal dataset from Mass General Brigham containing longer, more complex clinical narratives.

Key Insights and Ideas

  • Generalists Beat Specialists: Surprisingly, medical-specific models (Me-LLaMA, Meditron) performed significantly worse than advanced general-domain models. The authors attribute this to the specialists being built on older base models (like Llama 2) and lacking the sophisticated instruction-tuning found in newer general models like Llama 3.1.
  • The "Neither" Difficulty: Models struggled most with the "Neither" category—medications mentioned in a note that aren't currently being taken or stopped (e.g., "avoid ibuprofen" or "discussed starting Lisinopril"). This highlights a challenge in contextual nuance.
  • Hallucination Hurdles: In the complex MGB-Med dataset, Llama-3.1-70B showed a high "invalid" rate (170.9% relative to dataset size), frequently hallucinating medications that weren't in the text. GPT-4o was much more grounded, with an invalid rate of only 0.45%.
  • Few-Shot > CoT: While Chain-of-Thought is often a "magic bullet" for LLM performance, it provided inconsistent gains here. Providing five concrete examples (5-shot) was a more reliable way to improve performance and ensure the model followed the required output format.

Key Results

  • GPT-4o was the top performer overall, achieving an average F1-score of 94.0% for extraction and 72.7% for the end-to-end joint task in a zero-shot setting.
  • Open-Source Parity: Llama-3.1-70B-Instruct and Qwen2.5-72B-Instruct followed closely. In fact, Llama-3.1-70B actually outperformed GPT-4o on the joint task for the Re-CASI (76.2%) and MIV-Med (60.2%) datasets.
  • Scaling Matters: Across all model families, the larger versions (~70B parameters) consistently outperformed their smaller (~7B-14B) counterparts, confirming that model scale is a major driver of clinical reasoning capability.

Limitations and Future Work

The study primarily used "snippets" of text rather than full EHR notes. While this preserves local context, it might miss broader temporal cues found across an entire patient encounter.

The authors suggest that future work should explore Retrieval-Augmented Generation (RAG) to provide models with more medical knowledge and task-specific instruction tuning to reduce hallucinations and improve the handling of ambiguous "Neither" cases. Additionally, they noted that newer "reasoning" models (like DeepSeek-R1) were not included due to timing but represent a promising next step for error analysis.

Summary updated

Abstract

Identifying medication discontinuations in electronic health records (EHRs) is vital for patient safety but is often hindered by information being buried in unstructured notes. This study aims to evaluate the capabilities of advanced open-sourced and proprietary large language models (LLMs) in extracting medications and classifying their medication status from EHR notes, focusing on their scalability on medication information extraction without human annotation. We collected three EHR datasets from diverse sources to build the evaluation benchmark. We evaluated 12 advanced LLMs and explored multiple LLM prompting strategies. Performance on medication extraction, medication status classification, and their joint task (extraction then classification) was systematically compared across all experiments. We found that LLMs showed promising performance on the medication extraction and discontinuation classification from EHR notes. GPT-4o consistently achieved the highest average F1 scores in all tasks under zero-shot setting - 94.0% for medication extraction, 78.1% for discontinuation classification, and 72.7% for the joint task. Open-sourced models followed closely, Llama-3.1-70B-Instruct achieved the highest performance in medication status classification on the MIV-Med dataset (68.7%) and in the joint task on both the Re-CASI (76.2%) and MIV-Med (60.2%) datasets. Medical-specific LLMs demonstrated lower performance compared to advanced general-domain LLMs. Few-shot learning generally improved performance, while CoT reasoning showed inconsistent gains. LLMs demonstrate strong potential for medication extraction and discontinuation identification on EHR notes, with open-sourced models offering scalable alternatives to proprietary systems and few-shot can further improve LLMs' capability.
BibTeX
@article{shao2025scalable,
  author = {Chong Shao and Douglas Snyder and Chao Li and Bowen Gu and Kerry Ngan and Chun‐Ting Yang and Jiageng Wu and Richard Wyss and Kueiyu Joshua Lin and Jie Yang},
  title = {{Scalable medication extraction and discontinuation identification from electronic health records using large language models}},
  journal = {Journal of Clinical Epidemiology},
  year = {2025},
  doi = {10.1016/j.jclinepi.2025.112049},
}

Loading related content...

Figure