Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers

Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, Jimmy Huang

TL;DR

This paper evaluates the zero-shot capabilities of ChatGPT (gpt-3.5-turbo) on four core biomedical NLP tasks across 11 benchmark datasets. The authors discover that while ChatGPT lags behind domain-specific models fine-tuned on massive datasets, it frequently outperforms them in low-resource scenarios where training data is scarce or unavailable. This highlights ChatGPT's potential as an out-of-the-box tool for specialized biomedical applications without the need for costly data annotation.

What Problem Does This Paper Address?

Biomedical Natural Language Processing (BioNLP) tasks—such as extracting drug-to-drug interactions or summarizing medical dialogues—typically rely on specialized models like BioGPT or BioBART. While powerful, these models require extensive fine-tuning on large, domain-specific annotated datasets.

In the biomedical field, creating these datasets is a major bottleneck: annotation requires expensive expert knowledge and significant time. The authors investigate whether a general-purpose large language model (LLM) like ChatGPT can bypass this bottleneck by performing these highly specialized tasks effectively in a zero-shot setting (i.e., without any task-specific training data).

Methods

The authors evaluate ChatGPT in a strict zero-shot learning paradigm. For each test sample, they construct a prompt by combining a specific task instruction with the input text, providing no few-shot examples to guide the model.

They compare ChatGPT's performance against two state-of-the-art, domain-specifically pre-trained and fine-tuned baselines:

  • BioGPT: A decoder-only transformer model trained on PubMed titles and abstracts.
  • BioBART: An encoder-decoder sequence-to-sequence model also pre-trained on PubMed abstracts.

Additionally, the authors conduct prompt engineering experiments to analyze how varying the descriptiveness of instructions (such as including or omitting taxonomic definitions) impacts ChatGPT's performance.

Datasets

The evaluation spans 11 benchmark datasets covering four distinct tasks:

  • Relation Extraction: BC5CDR (Chemical-Disease), KD-DTI (Drug-Target), and DDI (Drug-Drug).
  • Document Classification: HoC (Hallmarks of Cancer).
  • Question Answering: PubMedQA.
  • Abstractive Summarization: iCliniq and HealthCareMagic (doctor-patient dialogues); MeQSum and MEDIQA-QS (medical questions); MEDIQA-MAS and MEDIQA-ANS (medical answers).

The specific prompts and dataset splits used for these tasks are detailed in the paper's reference tables:

Prompts and dataset details for Relation Extraction, Document Classification, and Question Answering tasks.
Prompts and dataset details for Abstractive Summarization tasks.

Key Insights and Ideas

  • The Low-Resource Advantage: ChatGPT’s zero-shot performance is highly dependent on the volume of training data available to the baseline models. When the baseline's training set is small (e.g., BC5CDR with 500 samples) or non-existent (e.g., the MEDIQA datasets), zero-shot ChatGPT frequently beats or matches the fine-tuned models. Conversely, when massive training sets are available (e.g., KD-DTI's 12,000 instances), fine-tuned models maintain a clear edge.
  • Extreme Sensitivity to Prompts: In specialized domains, prompt design is critical. Providing explicit definitions of terms or categories dramatically improves ChatGPT's accuracy. For example, adding definitions to the HoC classification prompt boosted the F1 score from 38.20 to 59.14.
  • High Recall, Low Precision: In relation extraction tasks, ChatGPT tends to generate long, descriptive answers. This verbosity leads to high recall (it catches most true relations) but lower precision (it introduces many false positives).

Key Results

  • Relation Extraction: On the small BC5CDR dataset, zero-shot ChatGPT achieved an F1 score of 48.42, outperforming fine-tuned BioGPT (46.17). On the DDI dataset, ChatGPT swept BioGPT across all metrics (F1 of 46.43 vs. 40.76). However, on the larger KD-DTI dataset, BioGPT won decisively (F1 of 38.42 vs. 29.74).
  • Document Classification & QA: Fine-tuned BioGPT significantly outperformed ChatGPT on HoC classification (F1 of 85.12 vs. 59.14) and PubMedQA (Accuracy of 78.20% vs. 51.60%), where BioGPT benefited from thousands of task-specific training pairs.
  • Summarization: For datasets lacking dedicated training data (MEDIQA-ANS and MEDIQA-MAS), ChatGPT outperformed both BioBART-Base and BioBART-Large across all ROUGE metrics. On MEDIQA-ANS, ChatGPT scored a ROUGE-1 of 28.7 compared to BioBART-Large's 21.6.

Limitations and Future Work

The authors note several limitations inherent to evaluating closed-source LLMs:

  • Reproducibility: Because ChatGPT is continuously updated and non-deterministic, its outputs can vary over time for the exact same prompt.
  • Data Contamination: Since ChatGPT’s exact pre-training corpus is unknown, it is difficult to guarantee that some evaluation data was not seen during its training phase.
  • Cost of Scale: While newer models like GPT-4 might offer superior accuracy, the authors point out that GPT-4 is roughly 60x more expensive than ChatGPT, making large-scale deployment costly.

In future work, the authors plan to expand their evaluation to more complex biomedical tasks, including Named Entity Recognition (NER), entity linking, and information retrieval, while also exploring ethical implications such as bias and privacy.

Summary updated

Abstract

ChatGPT is a large language model developed by OpenAI. Despite its impressive performance across various tasks, no prior work has investigated its capability in the biomedical domain yet. To this end, this paper aims to evaluate the performance of ChatGPT on various benchmark biomedical tasks, such as relation extraction, document classification, question answering, and summarization. To the best of our knowledge, this is the first work that conducts an extensive evaluation of ChatGPT in the biomedical domain. Interestingly, we find based on our evaluation that in biomedical datasets that have smaller training sets, zero-shot ChatGPT even outperforms the state-of-the-art fine-tuned generative transformer models, such as BioGPT and BioBART. This suggests that ChatGPT's pre-training on large text corpora makes it quite specialized even in the biomedical domain. Our findings demonstrate that ChatGPT has the potential to be a valuable tool for various tasks in the biomedical domain that lack large annotated data.
BibTeX
@inproceedings{jahan2023evaluation,
  author = {Israt Jahan and Md Tahmid Rahman Laskar and Chun Peng and Jimmy Huang},
  title = {{Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}

Loading related content...

Figure