Abstract
Medication information is crucial for clinical routine and research. However, a vast amount is stored in unstructured text, such as doctoral letters, requiring manual extraction—a resource-intensive, error-prone task. Automating this process comes with significant constraints in a clinical setup, including the demand for clinical expertise, limited time-resources, restricted IT infrastructure, and the demand for transparent predictions. Recent advances in generative large language models (LLMs) and parameter-efficient fine-tuning methods show potential to address these challenges. We evaluated local LLMs for end-to-end extraction of medication information, combining named entity recognition and relation extraction. We used format-restricting instructions and developed a feedback pipeline to facilitate automated evaluation. We applied token-level Shapley values to visualize and quantify token contributions, to improve transparency of model predictions. Two open-source LLMs—one general (Llama) and one domain-specific (OpenBioLLM)—were evaluated on the English n2c2 2018 corpus and the German CARDIO:DE corpus. OpenBioLLM frequently struggled with structured outputs and hallucinations. Fine-tuned Llama models achieved new state-of-the-art results, improving F1-score by 10% for adverse drug events and 6% for medication reasons on English data. On the German dataset, Llama established a new benchmark, outperforming traditional machine learning methods by 16% micro average F1-score. Our findings show that fine-tuned local open-source generative LLMs outperform SOTA methods for medication information extraction, delivering high performance with limited time and IT resources in a clinical setup.
Introduction: The Clinical Data Bottleneck
In the medical world, critical data—such as why a patient was prescribed a drug or what side effects they experienced—is often buried in unstructured "doctor’s letters." Manually extracting this information is a bottleneck: it is slow, expensive, and prone to error.
While LLMs seem like an obvious solution, the clinical environment imposes four harsh constraints: clinical expertise requirements, privacy concerns (prohibiting cloud-based APIs), limited IT infrastructure, and the need for transparency. Doctors won't trust a "black box" model; they need to know why a model made a specific prediction.
Methodology: An End-to-End Pipeline
The researchers developed an end-to-end (e2e) pipeline that performs Named Entity Recognition (NER) and Relation Extraction (RE) in a single step, producing structured JSON outputs.
Model Selection and Fine-Tuning
The study compared Llama 3.1 (8B and 70B) against OpenBioLLM, a domain-specific medical model. To ensure usable data, the models were forced to output structured JSON via Pydantic object definitions. The team employed Parameter-Efficient Fine-Tuning (PEFT) via LoRA and QLoRA, allowing them to train a 70B parameter model on a single NVIDIA H100 GPU—a feat achievable within many modern hospital IT infrastructures.
Evaluation and Interpretability
Recognizing that "exact matches" are a poor metric for clinical text (e.g., "qhs" vs. "at bedtime"), they used a larger model (Mistral-Large) to act as a "clinical judge," evaluating semantic similarity. To enhance transparency, they used Shapley values to visualize which specific words in the input text led to the model's output, enabling clinicians to better understand and trust predictions.
Key Results and Benchmarks
The study utilized two primary benchmarks: the English n2c2 2018 corpus and the German CARDIO:DE v1.1 corpus.
- New SOTA on English Data: The fine-tuned Llama 70B achieved a 93% micro-average F1-score on the n2c2 dataset. It specifically improved the extraction of difficult categories: Adverse Drug Events (+10%) and Medication Reasons (+6%) compared to previous best-in-class models.
- German Benchmark: On the CARDIO:DE dataset, Llama established a new benchmark with a 90% F1-score, outperforming traditional BERT-based methods by 16%.
- Generalist vs. Specialist: Surprisingly, the general-purpose Llama 3.1 outperformed the medical-specific OpenBioLLM, which frequently struggled with hallucinations and maintaining JSON structure.
Conclusion and Future Work
This research demonstrates that fine-tuned local open-source LLMs can deliver high performance with limited resources in a clinical setup. Future work includes building a graphical user interface for physicians that incorporates Shapley value "heatmaps," highlighting the evidence the AI used to suggest a medication change or flag a potential side effect.
Summary updated