TL;DR
This paper evaluates the zero-shot capabilities of ChatGPT (gpt-3.5-turbo) on four core biomedical NLP tasks across 11 benchmark datasets. The authors discover that while ChatGPT lags behind domain-specific models fine-tuned on massive datasets, it frequently outperforms them in low-resource scenarios where training data is scarce or unavailable. This highlights ChatGPT's potential as an out-of-the-box tool for specialized biomedical applications without the need for costly data annotation.
What Problem Does This Paper Address?
Biomedical Natural Language Processing (BioNLP) tasks—such as extracting drug-to-drug interactions or summarizing medical dialogues—typically rely on specialized models like BioGPT or BioBART. While powerful, these models require extensive fine-tuning on large, domain-specific annotated datasets.
In the biomedical field, creating these datasets is a major bottleneck: annotation requires expensive expert knowledge and significant time. The authors investigate whether a general-purpose large language model (LLM) like ChatGPT can bypass this bottleneck by performing these highly specialized tasks effectively in a zero-shot setting (i.e., without any task-specific training data).
Methods
The authors evaluate ChatGPT in a strict zero-shot learning paradigm. For each test sample, they construct a prompt by combining a specific task instruction with the input text, providing no few-shot examples to guide the model.
They compare ChatGPT's performance against two state-of-the-art, domain-specifically pre-trained and fine-tuned baselines:
- BioGPT: A decoder-only transformer model trained on PubMed titles and abstracts.
- BioBART: An encoder-decoder sequence-to-sequence model also pre-trained on PubMed abstracts.
Additionally, the authors conduct prompt engineering experiments to analyze how varying the descriptiveness of instructions (such as including or omitting taxonomic definitions) impacts ChatGPT's performance.
Datasets
The evaluation spans 11 benchmark datasets covering four distinct tasks:
- Relation Extraction: BC5CDR (Chemical-Disease), KD-DTI (Drug-Target), and DDI (Drug-Drug).
- Document Classification: HoC (Hallmarks of Cancer).
- Question Answering: PubMedQA.
- Abstractive Summarization: iCliniq and HealthCareMagic (doctor-patient dialogues); MeQSum and MEDIQA-QS (medical questions); MEDIQA-MAS and MEDIQA-ANS (medical answers).
The specific prompts and dataset splits used for these tasks are detailed in the paper's reference tables:
Key Insights and Ideas
- The Low-Resource Advantage: ChatGPT’s zero-shot performance is highly dependent on the volume of training data available to the baseline models. When the baseline's training set is small (e.g., BC5CDR with 500 samples) or non-existent (e.g., the MEDIQA datasets), zero-shot ChatGPT frequently beats or matches the fine-tuned models. Conversely, when massive training sets are available (e.g., KD-DTI's 12,000 instances), fine-tuned models maintain a clear edge.
- Extreme Sensitivity to Prompts: In specialized domains, prompt design is critical. Providing explicit definitions of terms or categories dramatically improves ChatGPT's accuracy. For example, adding definitions to the HoC classification prompt boosted the F1 score from 38.20 to 59.14.
- High Recall, Low Precision: In relation extraction tasks, ChatGPT tends to generate long, descriptive answers. This verbosity leads to high recall (it catches most true relations) but lower precision (it introduces many false positives).
Key Results
- Relation Extraction: On the small BC5CDR dataset, zero-shot ChatGPT achieved an F1 score of 48.42, outperforming fine-tuned BioGPT (46.17). On the DDI dataset, ChatGPT swept BioGPT across all metrics (F1 of 46.43 vs. 40.76). However, on the larger KD-DTI dataset, BioGPT won decisively (F1 of 38.42 vs. 29.74).
- Document Classification & QA: Fine-tuned BioGPT significantly outperformed ChatGPT on HoC classification (F1 of 85.12 vs. 59.14) and PubMedQA (Accuracy of 78.20% vs. 51.60%), where BioGPT benefited from thousands of task-specific training pairs.
- Summarization: For datasets lacking dedicated training data (MEDIQA-ANS and MEDIQA-MAS), ChatGPT outperformed both BioBART-Base and BioBART-Large across all ROUGE metrics. On MEDIQA-ANS, ChatGPT scored a ROUGE-1 of 28.7 compared to BioBART-Large's 21.6.
Limitations and Future Work
The authors note several limitations inherent to evaluating closed-source LLMs:
- Reproducibility: Because ChatGPT is continuously updated and non-deterministic, its outputs can vary over time for the exact same prompt.
- Data Contamination: Since ChatGPT’s exact pre-training corpus is unknown, it is difficult to guarantee that some evaluation data was not seen during its training phase.
- Cost of Scale: While newer models like GPT-4 might offer superior accuracy, the authors point out that GPT-4 is roughly 60x more expensive than ChatGPT, making large-scale deployment costly.
In future work, the authors plan to expand their evaluation to more complex biomedical tasks, including Named Entity Recognition (NER), entity linking, and information retrieval, while also exploring ethical implications such as bias and privacy.
Summary updated