TL;DR
This paper evaluates ChatGPT's zero-shot performance on several biomedical NLP tasks, finding it can outperform fine-tuned models on smaller datasets. This suggests ChatGPT's extensive pre-training makes it a powerful tool for biomedical applications where large annotated datasets are scarce.
What Problem Does This Paper Address?
The biomedical domain has seen significant advancements with specialized NLP models like BioBERT and BioGPT. However, these models typically require extensive fine-tuning on task-specific datasets, which are often difficult and time-consuming to create. This paper addresses the challenge of leveraging powerful language models like ChatGPT in the biomedical field without the need for such resource-intensive fine-tuning. Specifically, it investigates whether ChatGPT, in a zero-shot setting (i.e., without any task-specific training examples), can achieve competitive performance on key biomedical tasks.
Methods
The authors evaluate ChatGPT on four biomedical tasks: relation extraction, document classification, question answering, and abstractive summarization. For each task, they construct specific prompts that guide ChatGPT to perform the task based on the provided input text. These prompts are designed to be clear and informative, sometimes including definitions of entities or classes relevant to the task. ChatGPT's responses are then compared against gold-standard labels. The evaluation is conducted in a zero-shot manner, meaning ChatGPT is not provided with any examples from the specific biomedical datasets during its "training" for these tasks.
Datasets
The paper utilizes several established biomedical datasets for evaluation:
- Relation Extraction: BC5CDR (chemical-disease), KD-DTI (drug-target), and DDI (drug-drug).
- Document Classification: HoC (Hallmarks of Cancer corpus).
- Question Answering: PubMedQA.
- Abstractive Summarization: MeQSum, MEDIQA-QS, MEDIQA-ANS, MEDIQA-MAS, iCliniq, and HealthCareMagic.
No new benchmarks were introduced; the focus was on evaluating ChatGPT on existing, well-defined biomedical datasets.
Key Insights and Ideas
- Small Data Advantage: ChatGPT's zero-shot performance can surprisingly outperform state-of-the-art fine-tuned models (like BioGPT and BioBART) on biomedical datasets with smaller training set sizes. This highlights the power of large-scale pre-training.
- Prompt Sensitivity: ChatGPT's performance is notably sensitive to the way prompts are phrased. More descriptive prompts, especially those that clearly define the task and relevant entities or classes, tend to yield better results.
- Trade-offs in Relation Extraction: While ChatGPT achieves higher recall in relation extraction, it often sacrifices precision, leading to more verbose and sometimes inaccurate extractions compared to fine-tuned models.
- Potential for Low-Resource Scenarios: The findings strongly suggest that ChatGPT can be a valuable tool for biomedical tasks where obtaining large annotated datasets is a significant bottleneck.
Key Results
- Relation Extraction: ChatGPT achieved higher recall but lower precision than BioGPT on BC5CDR and KD-DTI. However, it outperformed BioGPT in the DDI dataset. Notably, on BC5CDR and DDI (datasets with smaller training sets), ChatGPT's zero-shot performance was competitive or superior.
- Document Classification: ChatGPT achieved an F1 score of 59.14 on the HoC dataset, while fine-tuned BioGPT achieved 85.12. Prompt variations significantly impacted ChatGPT's performance, with more descriptive prompts leading to better F1 scores.
- Question Answering: ChatGPT achieved 51.60% accuracy on PubMedQA, significantly lower than fine-tuned BioGPT (78.20%). However, it achieved over 50% accuracy without any task-specific examples.
- Abstractive Summarization: ChatGPT performed worse than BioBART on datasets with dedicated training sets. However, it performed comparably or better on datasets without dedicated training data (MEDIQA-QS, MEDIQA-ANS, MEDIQA-MAS), reinforcing its utility in low-resource settings.
Limitations and Future Work
Limitations: * The exact training data for ChatGPT is unknown, which could lead to potential overlap with evaluation datasets. * A newer, more expensive model (GPT-4) has been released, which might offer higher accuracy. * ChatGPT's responses can be non-reproducible, meaning the same prompt might yield different results over time or across different runs. * The cost and limited access to newer models like GPT-4 can be a barrier.
Future Work: * Investigate ChatGPT's performance on more challenging biomedical tasks like named entity recognition and entity linking. * Explore its capabilities in information retrieval. * Examine the ethical implications, such as bias and privacy concerns, of using ChatGPT in the biomedical domain.
Summary updated