TL;DR
This paper benchmarks 12 state-of-the-art Large Language Models (LLMs) on their ability to extract medications and identify discontinuations from unstructured clinical notes. The researchers found that while GPT-4o leads the pack, top-tier open-source models like Llama-3.1-70B and Qwen2.5-72B are now highly competitive, offering a scalable, privacy-conscious path for automating drug-safety surveillance and clinical decision support.
What Problem Does This Paper Address?
Electronic Health Records (EHRs) are filled with critical patient data, but much of it is "buried" in unstructured clinical notes. Specifically, identifying when a patient stops taking a medication is vital for preventing adverse drug events and understanding treatment failures.
Standard structured data (like insurance claims) often misses these nuances—claims show when a prescription is filled, but not necessarily when a doctor tells a patient to stop taking it due to a side effect. Manually extracting this information is too slow and expensive to scale. This paper investigates whether LLMs can bridge this gap by accurately identifying medication names and their status (active, discontinued, or neither) without requiring massive human-annotated training sets.
Methods
The authors conducted a systematic benchmark of 12 LLMs across three primary tasks:
- Medication Extraction: Identifying drug names in text (Named Entity Recognition).
- Isolated Status Classification: Given a drug name, classifying it as "active," "discontinued," or "neither" based on context.
- Joint Task (End-to-End): Extracting the drug and classifying its status in one go—the most realistic "real-world" scenario.
They tested three prompting strategies: Zero-shot (no examples), 5-shot (five examples provided), and Chain-of-Thought (CoT) (asking the model to "think step-by-step"). The model lineup included proprietary giants (GPT-4o), general-purpose open-source models (Llama 3.1, Qwen 2.5, Mistral), and medical-specific models (Me-LLaMA, Meditron).
Datasets
The study utilized three distinct datasets to ensure robustness:
- Re-CASI: A publicly available, de-identified dataset derived from the Clinical Acronym Sense Inventory.
- MIV-Med (New): A newly annotated dataset created by the authors using discharge summaries from the public MIMIC-IV database. It was annotated by experts, including an MD.
- MGB-Med (New): An internal dataset from Mass General Brigham containing longer, more complex clinical narratives.
Key Insights and Ideas
- Generalists Beat Specialists: Surprisingly, medical-specific models (Me-LLaMA, Meditron) performed significantly worse than advanced general-domain models. The authors attribute this to the specialists being built on older base models (like Llama 2) and lacking the sophisticated instruction-tuning found in newer general models like Llama 3.1.
- The "Neither" Difficulty: Models struggled most with the "Neither" category—medications mentioned in a note that aren't currently being taken or stopped (e.g., "avoid ibuprofen" or "discussed starting Lisinopril"). This highlights a challenge in contextual nuance.
- Hallucination Hurdles: In the complex MGB-Med dataset, Llama-3.1-70B showed a high "invalid" rate (170.9% relative to dataset size), frequently hallucinating medications that weren't in the text. GPT-4o was much more grounded, with an invalid rate of only 0.45%.
- Few-Shot > CoT: While Chain-of-Thought is often a "magic bullet" for LLM performance, it provided inconsistent gains here. Providing five concrete examples (5-shot) was a more reliable way to improve performance and ensure the model followed the required output format.
Key Results
- GPT-4o was the top performer overall, achieving an average F1-score of 94.0% for extraction and 72.7% for the end-to-end joint task in a zero-shot setting.
- Open-Source Parity: Llama-3.1-70B-Instruct and Qwen2.5-72B-Instruct followed closely. In fact, Llama-3.1-70B actually outperformed GPT-4o on the joint task for the Re-CASI (76.2%) and MIV-Med (60.2%) datasets.
- Scaling Matters: Across all model families, the larger versions (~70B parameters) consistently outperformed their smaller (~7B-14B) counterparts, confirming that model scale is a major driver of clinical reasoning capability.
Limitations and Future Work
The study primarily used "snippets" of text rather than full EHR notes. While this preserves local context, it might miss broader temporal cues found across an entire patient encounter.
The authors suggest that future work should explore Retrieval-Augmented Generation (RAG) to provide models with more medical knowledge and task-specific instruction tuning to reduce hallucinations and improve the handling of ambiguous "Neither" cases. Additionally, they noted that newer "reasoning" models (like DeepSeek-R1) were not included due to timing but represent a promising next step for error analysis.
Summary updated