Abstract
Generalist foundation models such as GPT-4 have displayed surprising capabilities in a wide variety of domains and tasks. Yet, there is a prevalent assumption that they cannot match specialist capabilities of fine-tuned models. For example, most explorations to date on medical competency benchmarks have leveraged domain-specific training, as exemplified by efforts on BioGPT and Med-PaLM. We build on a prior study of GPT-4's capabilities on medical challenge benchmarks in the absence of special training. Rather than using simple prompting to highlight the model's out-of-the-box capabilities, we perform a systematic exploration of prompt engineering. We find that prompting innovation can unlock deeper specialist capabilities and show that GPT-4 easily tops prior leading results for medical benchmarks. The prompting methods we explore are general purpose, and make no specific use of domain expertise, removing the need for expert-curated content. Our experimental design carefully controls for overfitting during the prompt engineering process. We introduce Medprompt, based on a composition of several prompting strategies. With Medprompt, GPT-4 achieves state-of-the-art results on all nine of the benchmark datasets in the MultiMedQA suite. The method outperforms leading specialist models such as Med-PaLM 2 by a significant margin with an order of magnitude fewer calls to the model. Steering GPT-4 with Medprompt achieves a 27% reduction in error rate on the MedQA dataset over the best methods to date achieved with specialist models and surpasses a score of 90% for the first time. Beyond medical problems, we show the power of Medprompt to generalize to other domains and provide evidence for the broad applicability of the approach via studies of the strategy on exams in electrical engineering, machine learning, philosophy, accounting, law, nursing, and clinical psychology.
Introduction
In the artificial intelligence community, a long-standing paradigm suggests that achieving expert-level performance in specialized domains—such as medicine, law, or engineering—necessitates intensive, domain-specific fine-tuning. Models like BioGPT and Med-PaLM 2 have historically validated this assumption, relying on expensive training cycles and expert-curated datasets to instill specialized knowledge. However, the emergence of massive generalist foundation models like GPT-4 raises a provocative question: Is specialized training a requirement for expertise, or can the latent knowledge within a generalist model be effectively "steered" to match or exceed specialist performance?
This study investigates the potential of advanced prompt engineering to unlock specialist capabilities in GPT-4. By introducing a systematic framework called Medprompt, we demonstrate that a generalist model can outperform the most advanced fine-tuned medical models without any domain-specific training.
The Medprompt Framework
Medprompt is a composition of three general-purpose prompting strategies designed to maximize the reasoning and retrieval capabilities of GPT-4. Notably, these strategies require zero medical expertise to implement, relying instead on the model's inherent logic and semantic understanding.
1. Dynamic Few-shot Selection
Traditional few-shot prompting uses a fixed set of examples for every query. Medprompt improves upon this by using a k-Nearest Neighbors (k-NN) approach. Using an embedding model (text-embedding-ada-002), the system identifies training questions that are semantically similar to the target test question. This provides the model with the most relevant context, effectively acting as a dynamic, in-context retrieval mechanism.
2. Self-Generated Chain of Thought (CoT)
Chain of Thought prompting encourages models to "think out loud" before providing an answer. Rather than relying on human experts to write these reasoning steps—which can be time-consuming and may not align with the model's internal logic—Medprompt tasks GPT-4 with generating its own CoT for training examples. The framework only retains reasoning chains that lead to the correct answer, creating a high-quality library of logic for the model to emulate during testing.
3. Choice Shuffling Ensemble
To address "position bias," where models may disproportionately favor certain multiple-choice options (e.g., always choosing 'A'), Medprompt employs a shuffling ensemble. The order of answers is randomized across multiple runs, and a majority vote determines the final output. This technique significantly increases the robustness of the model's predictions.
Experimental Results
The researchers evaluated Medprompt primarily on the MultiMedQA suite, which includes the USMLE-style MedQA, as well as MedMCQA and PubMedQA. To ensure the integrity of the results, the team employed a rigorous "eyes-off" methodology, holding out 20% of the data to prevent overfitting the prompts to the test set.
Breaking the 90% Barrier
GPT-4 steered by Medprompt achieved a record-breaking 90.2% on the MedQA (USMLE) dataset. This represents the first time any AI model has surpassed the 90% threshold on this benchmark, achieving a 27% reduction in error rate compared to the previous state-of-the-art specialist model, Med-PaLM 2.
Cross-Domain Generalization
While named for its medical application, the Medprompt strategy is domain-agnostic. When tested on non-medical datasets from the MMLU (covering Law, Accounting, and Philosophy) and the NCLEX nursing exam, Medprompt provided an average improvement of +7.3% over zero-shot baselines. This suggests that the framework is a general-purpose "meta-strategy" for expert-level tasks.
Discussion and Conclusion
The success of Medprompt challenges the necessity of intensive fine-tuning for specialized tasks. The study reveals that GPT-4's self-generated reasoning chains often outperform those written by human medical experts, likely because the model's internal logic is better suited to its own processing strengths. Furthermore, Medprompt is highly efficient; while Med-PaLM 2 required an ensemble of 44 model calls per question, Medprompt achieved superior results with only 5 calls.
Despite these achievements, limitations remain. High performance on multiple-choice benchmarks does not equate to real-world clinical safety, and the risk of hallucinations persists. Future work will explore adapting these ensemble strategies for open-ended generation and investigating the potential synergies between advanced prompting and fine-tuning.
Summary updated