Abstract
Identifying medication discontinuations in electronic health records (EHRs) is vital for patient safety but is often hindered by information being buried in unstructured notes. This study aims to evaluate the capabilities of advanced open-sourced and proprietary large language models (LLMs) in extracting medications and classifying their medication status from EHR notes, focusing on their scalability on medication information extraction without human annotation. We collected three EHR datasets from diverse sources to build the evaluation benchmark. We evaluated 12 advanced LLMs and explored multiple LLM prompting strategies. Performance on medication extraction, medication status classification, and their joint task (extraction then classification) was systematically compared across all experiments. We found that LLMs showed promising performance on the medication extraction and discontinuation classification from EHR notes. GPT-4o consistently achieved the highest average F1 scores in all tasks under zero-shot setting - 94.0% for medication extraction, 78.1% for discontinuation classification, and 72.7% for the joint task. Open-sourced models followed closely, Llama-3.1-70B-Instruct achieved the highest performance in medication status classification on the MIV-Med dataset (68.7%) and in the joint task on both the Re-CASI (76.2%) and MIV-Med (60.2%) datasets. Medical-specific LLMs demonstrated lower performance compared to advanced general-domain LLMs. Few-shot learning generally improved performance, while CoT reasoning showed inconsistent gains. LLMs demonstrate strong potential for medication extraction and discontinuation identification on EHR notes, with open-sourced models offering scalable alternatives to proprietary systems and few-shot can further improve LLMs' capability.
Introduction
Electronic Health Records (EHRs) are filled with critical patient data, but much of it is "buried" in unstructured clinical notes. Specifically, identifying when a patient stops taking a medication is vital for preventing adverse drug events and understanding treatment failures. Standard structured data, such as insurance claims, often misses these nuances—claims show when a prescription is filled, but not necessarily when a doctor instructs a patient to stop due to side effects.
Manually extracting this information is resource-intensive and difficult to scale. This paper investigates whether Large Language Models (LLMs) can bridge this gap by accurately identifying medication names and their status (active, discontinued, or neither) without requiring massive human-annotated training sets.
Methodology
The authors conducted a systematic benchmark of 12 LLMs across three primary tasks: medication extraction (Named Entity Recognition), isolated status classification, and a joint end-to-end task that combines both extraction and classification.
Three prompting strategies were tested: Zero-shot, 5-shot (providing five examples), and Chain-of-Thought (CoT). The model lineup included proprietary systems (GPT-4o), general-purpose open-source models (Llama 3.1, Qwen 2.5, Mistral), and medical-specific models (Me-LLaMA, Meditron).
Datasets
The study utilized three distinct datasets to ensure robustness:
- Re-CASI: A publicly available, de-identified dataset derived from the Clinical Acronym Sense Inventory.
- MIV-Med: A newly annotated dataset created by the authors using discharge summaries from the public MIMIC-IV database, verified by medical experts.
- MGB-Med: An internal dataset from Mass General Brigham containing longer, more complex clinical narratives.
Key Results and Insights
Model Performance
GPT-4o was the top performer overall, achieving an average F1-score of 94.0% for extraction and 72.7% for the joint task in a zero-shot setting. However, top-tier open-source models like Llama-3.1-70B-Instruct and Qwen2.5-72B-Instruct followed closely. Notably, Llama-3.1-70B outperformed GPT-4o on the joint task for the Re-CASI (76.2%) and MIV-Med (60.2%) datasets.
Generalists vs. Specialists
Surprisingly, medical-specific models (Me-LLaMA, Meditron) performed significantly worse than advanced general-domain models. This suggests that the sophisticated instruction-tuning found in newer general models like Llama 3.1 outweighs the domain-specific pre-training of older specialized models.
Challenges in Context and Hallucination
Models struggled most with the "Neither" category—medications mentioned in a note that are neither active nor stopped (e.g., "avoid ibuprofen"). Additionally, in complex narratives, some open-source models showed higher hallucination rates; for instance, Llama-3.1-70B occasionally identified medications not present in the text, whereas GPT-4o remained highly grounded with an invalid rate of only 0.45%.
Conclusion
LLMs demonstrate strong potential for automating medication extraction and discontinuation identification from EHR notes. While proprietary models lead in grounding, open-source models offer a scalable and privacy-conscious alternative for clinical decision support. Future work will explore Retrieval-Augmented Generation (RAG) and task-specific instruction tuning to further improve the handling of ambiguous clinical contexts.
Summary updated