Abstract
Tumor necrosis factor inhibitors (TNFi) are a cornerstone in the treatment of autoimmune conditions, yet many patients eventually switch medications due to loss of efficacy, adverse events, or cost. Understanding these switching trajectories is vital for clinical research but traditionally requires labor-intensive manual chart reviews. This study evaluates the ability of large language models (LLMs) to automate this process using de-identified electronic health record (EHR) data from 9,187 patients at the University of California, San Francisco (2012–2023). We compared GPT-4 against eight open-source models to extract medication changes and the underlying reasons for switching. GPT-4 achieved micro-F1 scores of up to 0.83, while smaller models like Llama-3-8B and Starling-7B demonstrated competitive performance. These findings suggest that LLMs can effectively perform complex clinical reasoning tasks, offering a scalable alternative to manual chart review in pharmacoepidemiology.
Introduction
In the management of autoimmune diseases such as Rheumatoid Arthritis (RA) and Inflammatory Bowel Disease (IBD), TNFi biologics are frequently prescribed. However, treatment pathways are rarely linear. Patients often cycle through multiple TNFi agents because of "secondary failure" (loss of efficacy over time), side effects, or changes in insurance coverage.
While structured EHR data—such as medication orders and pharmacy claims—can confirm that a switch occurred, they rarely capture why. Uncovering the rationale behind a clinical decision typically requires a physician to manually review unstructured clinical notes, a process that is prohibitively expensive and time-consuming at scale. This study investigates whether LLMs can bridge this gap by automating the extraction of switching reasons and medication trajectories directly from clinical narratives.
Methods
Dataset and Cohort
The researchers utilized the UCSF Information Commons, a longitudinal EHR dataset. The initial cohort included 12,442 unique patients with over 190,000 TNFi medication orders. After filtering for patients with at least six months of follow-up, the final study group consisted of 9,187 patients. Within this group, 3,104 switches were identified across 2,112 unique patients.
LLM Extraction Approach
The study employed a zero-shot extraction framework, where models were not fine-tuned on the specific dataset but were instead provided with structured prompts.
- Prompt Engineering: Four prompt variations were tested. The most effective prompts provided the models with explicit categorical reasons for switching (e.g., "Adverse event," "Insurance/Cost," "Lack of efficacy").
- Models Evaluated: The proprietary GPT-4-turbo-128k was benchmarked against eight open-source models, including Llama-3-8B-Instruct, Starling-7B-beta, and BioMistral-7B.
- Validation: Performance was validated against a "Gold Standard" of manual annotations by a physician and a "Silver Standard" of automated labels from structured data.
Key Results
Model Performance
GPT-4 emerged as the most consistent performer, achieving micro-F1 scores of 0.75 for identifying the stopped medication, 0.80 for the started medication, and 0.83 for the reason for switching.
Surprisingly, smaller open-source models (7B–8B parameters) proved highly capable. Starling-7B-beta and Llama-3-8B were the most competitive, with Starling-7B-alpha actually outperforming GPT-4 in identifying the specific TNFi that was discontinued (F1 of 0.897). However, these smaller models generally struggled more than GPT-4 with the nuanced reasoning required to categorize the reason for the switch.
Clinical Insights
The extraction revealed the primary drivers of medication switching in the cohort:
- Lack of Efficacy: 56.9%
- Adverse Events: 13.5%
- Insurance/Cost: 10.8%
Analysis of the trajectories showed that 16.1% of the cohort switched at least once, while a small but significant subset (1.6%) switched three or more times, highlighting a "difficult-to-treat" population that may require more intensive clinical intervention.
Discussion
The Value of Unstructured Text
A key finding of this study is that structured data is often a poor proxy for clinical reality. GPT-4’s performance was significantly higher when extracting from clinical notes than when compared against structured medication orders. This confirms that vital context—such as patient preference or subtle side effects—is often only captured in the narrative text of a clinical encounter.
Open-Source Viability
The competitiveness of 7B and 8B parameter models suggests a tiered approach for future clinical research. Researchers could utilize smaller, locally deployable models for high-volume tasks like drug identification, which preserves data privacy and reduces costs, while reserving larger models or human experts for complex reasoning tasks like determining the specific cause of treatment failure.
Limitations
This study was conducted at a single academic medical center (UCSF), which may limit the generalizability of the findings due to specific regional charting styles. Additionally, the models only analyzed the first clinical note associated with a switch, potentially missing nuances discussed in prior or subsequent visits. The study also did not account for patients who discontinued therapy entirely without switching to a new TNFi.
Conclusion
LLMs demonstrate significant potential in automating the extraction of complex clinical trajectories from EHR data. By accurately identifying why patients switch medications, these models can provide researchers with a more granular understanding of drug performance and patient outcomes in the real world. Future work should focus on expanding this methodology to other medication classes and multi-center datasets to further validate the utility of LLMs in automated chart summarization and pharmacoepidemiology.
Summary updated