PubMedQA: A Dataset for Biomedical Research Question Answering

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, Xinghua Lu

Abstract

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, Xinghua Lu. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

Building intelligent systems capable of reasoning over natural language is a primary goal of NLP, typically measured through Question Answering (QA) benchmarks. However, in the biomedical domain, existing datasets often lack the complexity required for deep scientific reasoning. To address this, the authors introduce PubMedQA, a novel dataset where models must answer research questions with "yes," "no," or "maybe" based on PubMed abstracts. Unlike general-domain datasets, PubMedQA demands high-level reasoning over quantitative scientific data and experimental results.

The PubMedQA Challenge

The authors identified a unique structural pattern in scientific literature: many PubMed articles use a research question as their title, and the abstract's conclusion directly answers that question. The body of the abstract provides the necessary context and methodology. By leveraging this structure, PubMedQA creates a task that specifically tests a model's ability to perform scientific reasoning—such as comparing experimental and control groups—to arrive at a definitive conclusion.

Methods

To establish a strong baseline, the authors utilize BioBERT, a language model pre-trained on large-scale biomedical corpora. They introduce two key strategies to maximize performance given limited expert-annotated data:

  1. Multi-phase Fine-tuning: The model is trained in three stages. It begins with pre-training on a massive, noisily-generated artificial dataset (Phase I), followed by fine-tuning on a bootstrapped unlabeled dataset using pseudo-labels (Phase II), and finally fine-tuning on a small, high-quality expert-annotated dataset (Phase III).
  2. Long Answer Auxiliary Supervision: During training, the model is tasked with predicting the Bag-of-Words (BoW) statistics of the abstract's actual conclusion (the "long answer") alongside the yes/no/maybe label. This auxiliary task acts as a regularizer, forcing the model's internal representations to capture the semantic essence of the conclusion.

Datasets

PubMedQA is divided into three distinct subsets to support the multi-phase training strategy:

The three subsets of PubMedQA: PQA-Artificial, PQA-Unlabeled, and PQA-Labeled, showing their respective sizes and internal structures.
  • PQA-Labeled (PQA-L): 1,000 instances manually annotated by medical experts for testing and validation.
  • PQA-Unlabeled (PQA-U): 61.2k instances with question titles and contexts, used for pseudo-labeling.
  • PQA-Artificial (PQA-A): 211.3k instances automatically generated by converting declarative titles into questions.

Key Insights

Analysis of the dataset reveals that scientific QA is uniquely suited for yes/no formats, with over 50% of PubMed question titles being answerable this way. Furthermore, quantitative reasoning is central to the task: 96.5% of questions require reasoning over quantitative contents, and 57.5% involve inter-group comparisons between experimental arms.

A Sankey diagram illustrating the relationship between question types, reasoning types (like inter-group comparison), and the presence of numerical interpretations in the context.

Results and Performance

The multi-phase fine-tuned BioBERT model, utilizing auxiliary long-answer supervision, achieved 68.1% accuracy. While this significantly outperforms simple baselines (such as the 55.2% majority guess), it still lags behind the 78.0% accuracy achieved by human experts. Ablation studies confirmed that the auxiliary BoW supervision improved results in the majority of experimental setups.

Limitations and Future Work

A significant hurdle remains in numerical interpretation; approximately 21% of contexts contain raw statistics (e.g., "p<0.05") without explicit natural language descriptions. Current models struggle to interpret these raw numbers. Future research will focus on improved numerical representation techniques and more complex auxiliary tasks, such as full long-answer generation, to further bridge the gap to human-level scientific reasoning.

Summary updated

BibTeX
@inproceedings{jin2019pubmedqa,
  author = {Qiao Jin and Bhuwan Dhingra and Zhengping Liu and William W. Cohen and Xinghua Lu},
  title = {{PubMedQA: A Dataset for Biomedical Research Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=21K2CeIJ8FZ},
}
Figure