Abstract
The emerging large language models (LLMs) are actively evaluated in various fields including healthcare. Most studies have focused on established benchmarks and standard parameters; however, the variation and impact of prompt engineering and fine-tuning strategies have not been fully explored. This study benchmarks GPT-3.5 Turbo, GPT-4, and Llama-7B against BERT models and medical fellows' annotations in identifying patients with metastatic cancer from discharge summaries. Results revealed that clear, concise prompts incorporating reasoning steps significantly enhanced performance. GPT-4 exhibited superior performance among all models. Notably, one-shot learning and fine-tuning provided no incremental benefit. The model's accuracy sustained even when keywords for metastatic cancer were removed or when half of the input tokens were randomly discarded. These findings underscore GPT-4's potential to substitute specialized models, such as PubMedBERT, through strategic prompt engineering, and suggest opportunities to improve open-source models, which are better suited to use in clinical settings.
Introduction
Clinical discharge summaries are a critical source of patient data, yet they are often unstructured and filled with complex medical jargon. Accurately identifying conditions like metastatic cancer is essential for both clinical care and research, but Electronic Health Records (EHRs) frequently lack precise labels. Traditionally, this task required specialized models like PubMedBERT, which necessitate extensive domain-specific fine-tuning. This study investigates whether general-purpose Large Language Models (LLMs) like GPT-4 can match or exceed the performance of these specialized models through strategic prompt engineering rather than traditional fine-tuning.
Methods and Datasets
The researchers utilized 1,873 discharge summaries from the MIMIC-III database. To establish a "gold standard," three medical fellows manually annotated 188 summaries, achieving a 94% agreement rate. The study compared three primary approaches:
- Zero-Shot Prompting: Testing six prompt variations, including simple instructions and structured "Chain-of-Thought" (CoT) prompts.
- One-Shot Learning: Providing the model with a single labeled example.
- Fine-Tuning: Training an open-source Llama-7B model using Low-Rank Adaptation (LoRA).
Robustness was evaluated by varying model temperature, removing key diagnostic terms, and randomly discarding up to 80% of the text (sparsity).
Key Results
GPT-4 emerged as the top performer, significantly outperforming specialized models and human experts. Using a specific prompt (Prompt 4) and a 3000-token window, GPT-4 achieved an F1 score of 0.941 and 100% precision. This surpassed the human fellows' F1 scores (0.710–0.774) and the domain-specific PubMedBERT (0.690).
The data indicates that prompt structure is a decisive factor. For instance, Prompt 4—which instructed the model to default to "No" when evidence was ambiguous—yielded the highest accuracy. Interestingly, for GPT-4, adding examples (one-shot) or explicit reasoning steps did not provide additional benefits, suggesting its internal reasoning capabilities are already highly optimized for this complexity.
Clinical Reasoning and Robustness
A standout finding was GPT-4's ability to infer clinical context without relying on specific keywords. Even when terms like "cancer" and "metastatic" were redacted, the model correctly identified cases by analyzing secondary indicators such as palliative care mentions or multi-organ involvement.
Furthermore, the model demonstrated extreme robustness to data loss, maintaining stable performance even when 40% of the tokens were randomly removed from the clinical notes. This suggests that LLMs can effectively navigate the "noisy" nature of real-world clinical documentation.
Discussion and Conclusion
While GPT-4 demonstrated superior reasoning, the study highlighted a significant performance gap with smaller open-source models like Llama-7B, even after fine-tuning. This suggests that for complex clinical tasks, model scale and general reasoning currently outweigh specialized training on smaller architectures. However, because proprietary models like GPT-4 present privacy challenges for clinical deployment, improving the prompting strategies and reasoning of open-source models remains a priority. Ultimately, these findings suggest that strategic prompt engineering can transform general-purpose LLMs into powerful tools for clinical information extraction, potentially replacing the need for labor-intensive, specialized model training.
Summary updated