PubMedQA: A Dataset for Biomedical Research Question Answering

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, Xinghua Lu

Abstract

We introduce PubMedQA, a novel biomedical question answering (QA) dataset collected from PubMed abstracts. The task of PubMedQA is to answer research questions with yes/no/maybe (e.g.: Do preoperative statins reduce atrial fibrillation after coronary artery bypass grafting?) using the corresponding abstracts. PubMedQA has 1k expert-annotated, 61.2k unlabeled and 211.3k artificially generated QA instances. Each PubMedQA instance is composed of (1) a question which is either an existing research article title or derived from one, (2) a context which is the corresponding abstract without its conclusion, (3) a long answer, which is the conclusion of the abstract and, presumably, answers the research question, and (4) a yes/no/maybe answer which summarizes the conclusion. PubMedQA is the first QA dataset where reasoning over biomedical research texts, especially their quantitative contents, is required to answer the questions. Our best performing model, multi-phase fine-tuning of BioBERT with long answer bag-of-word statistics as additional supervision, achieves 68.1% accuracy, compared to single human performance of 78.0% accuracy and majority-baseline of 55.2% accuracy, leaving much room for improvement.

Introduction

Building intelligent systems capable of reasoning over medical literature is a cornerstone of evidence-based medicine. However, existing biomedical QA datasets often fall into two categories: expertly annotated but too small for deep learning (e.g., BioASQ), or large but focused on simple "factoid" extraction that requires little logical synthesis.

To address this, we present PubMedQA, a dataset designed to test a model's ability to reason over complex, quantitative clinical text. Answering real-world research questions—such as whether a specific therapy is effective—requires a model to parse methodologies, compare experimental groups, and interpret statistical results. PubMedQA bridges the gap between simple extraction and complex scientific reasoning.

The PubMedQA Dataset

PubMedQA is constructed directly from PubMed abstracts. The dataset is uniquely structured: the article's title serves as the Question, the abstract body (excluding the conclusion) serves as the Context, and the abstract's conclusion serves as the Long Answer.

The structure of the three PubMedQA subsets: Artificial, Unlabeled, and Labeled.

The dataset is divided into three distinct subsets to facilitate different training paradigms:

  • PQA-L (Labeled): 1,000 instances expertly annotated with yes/no/maybe answers by medical researchers. This serves as the gold standard for evaluation.
  • PQA-U (Unlabeled): 61.2k instances of naturally occurring question-title abstracts that lack explicit yes/no/maybe labels.
  • PQA-A (Artificial): 211.3k instances automatically generated by converting declarative article titles into questions using simple linguistic heuristics.

Analysis of the dataset reveals its complexity: 96.5% of questions require reasoning over quantitative content, and 57.5% specifically require comparing multiple groups (e.g., experimental vs. control). Furthermore, about 21% of abstracts contain raw statistical numbers (e.g., "p < 0.05") without explicit natural language interpretations, posing a significant challenge for current language models.

Methodology: Multi-Phase Fine-Tuning

To maximize the utility of the limited expert-labeled data, we propose a multi-phase fine-tuning schedule using BioBERT. This approach leverages the larger, noisier subsets to prime the model before final training on the high-quality labeled data.

The multi-phase fine-tuning strategy, showing the progression from artificial data to pseudo-labeling and final expert-labeled supervision.

The training process follows three distinct phases:

  1. Phase I (Pre-training): The model is first trained on the massive PQA-A dataset. Despite the noise and label imbalance (92.8% "yes"), this phase provides a critical foundation.
  2. Phase II (Bootstrapping): The model pseudo-labels the PQA-U corpus, fine-tuning itself on its most confident predictions to improve domain adaptation.
  3. Final Phase: The model is fine-tuned on the small, high-quality PQA-L dataset.

Auxiliary Supervision: A key innovation in our approach is the use of the "Long Answer" (the abstract's conclusion) as an auxiliary training signal. During training, the model is tasked with predicting the binary Bag-of-Words (BoW) statistics of the conclusion alongside the yes/no/maybe label. This forces the model to capture the semantic essence of the conclusion within its internal representations, even though the conclusion is hidden during testing.

Results and Discussion

Our best-performing model—multi-phase fine-tuned BioBERT with auxiliary BoW supervision—achieved 68.1% accuracy and a 52.7% macro-F1 score. While this significantly outperforms the majority-class baseline of 55.2%, it remains well below the single human performance of 78.0% accuracy.

Ablation studies confirmed the efficacy of our training strategy: the multi-phase schedule strictly outperformed single-phase training, and the auxiliary BoW supervision improved results in 28 out of 40 experimental setups.

Despite these gains, the 10% gap between machine and human performance highlights the difficulty of biomedical reasoning. Future work may involve more sophisticated auxiliary tasks, such as abstractive summarization of the long answer, or developing architectures better suited for raw numerical reasoning. PubMedQA provides a challenging benchmark to drive these advancements in biomedical NLP.

Summary updated

Abstract

We introduce PubMedQA, a novel biomedical question answering (QA) dataset collected from PubMed abstracts. The task of PubMedQA is to answer research questions with yes/no/maybe (e.g.: Do preoperative statins reduce atrial fibrillation after coronary artery bypass grafting?) using the corresponding abstracts. PubMedQA has 1k expert-annotated, 61.2k unlabeled and 211.3k artificially generated QA instances. Each PubMedQA instance is composed of (1) a question which is either an existing research article title or derived from one, (2) a context which is the corresponding abstract without its conclusion, (3) a long answer, which is the conclusion of the abstract and, presumably, answers the research question, and (4) a yes/no/maybe answer which summarizes the conclusion. PubMedQA is the first QA dataset where reasoning over biomedical research texts, especially their quantitative contents, is required to answer the questions. Our best performing model, multi-phase fine-tuning of BioBERT with long answer bag-of-word statistics as additional supervision, achieves 68.1% accuracy, compared to single human performance of 78.0% accuracy and majority-baseline of 55.2% accuracy, leaving much room for improvement. PubMedQA is publicly available at https://pubmedqa.github.io.
BibTeX
@inproceedings{jin2019pubmedqa,
  author = {Qiao Jin and Bhuwan Dhingra and Zhengping Liu and William W. Cohen and Xinghua Lu},
  title = {{PubMedQA: A Dataset for Biomedical Research Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=21K2CeIJ8FZ},
}
Figure