A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature

Benjamin E. Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain James Marshall, Ani Nenkova, Byron C. Wallace

Introduction

Evidence-Based Medicine (EBM) is the cornerstone of modern clinical decision-making, requiring practitioners to synthesize the totality of available research. However, the exponential growth of medical literature—with approximately 100 new randomized controlled trial (RCT) manuscripts published daily—makes manual synthesis increasingly untenable. Natural Language Processing (NLP) offers a potential solution through automated information extraction. To facilitate this, we present EBM-NLP, a comprehensive dataset of 5,000 medical abstracts annotated with PICO (Participants, Interventions, Comparators, and Outcomes) elements. This corpus addresses the critical need for large-scale, high-quality human-annotated data to train robust medical NLP models.

Methodology

To balance scale, cost, and accuracy, a hybrid crowdsourcing strategy was implemented:

  1. Two-Stage Annotation: Tasks were decomposed into broad span identification followed by granular labeling (e.g., distinguishing age from condition within a participant description).
  2. Probabilistic Aggregation: Recognizing the technical nature of medical text, labels from Amazon Mechanical Turk (AMT) workers were aggregated using HMMCrowd, a probabilistic model that accounts for individual worker reliability.
  3. Expert Validation: A "gold standard" test set of 200 abstracts was independently annotated by medical professionals (MDs and medical students) to evaluate model and crowd performance.
  4. Vocabulary Mapping: Identified entities were mapped to the Medical Subject Headings (MeSH) vocabulary to support structured database integration.

The EBM-NLP Corpus

The EBM-NLP corpus consists of approximately 5,000 RCT abstracts sourced from PubMed, primarily covering cardiovascular disease, cancer, and autism. Unlike previous "flat" datasets, EBM-NLP features a hierarchical annotation scheme, particularly for Outcomes.

Figure 2: The hierarchical structure of Outcome labels, categorizing results into physical, mental, and non-health domains.

This hierarchy allows for nuanced information extraction, moving beyond simple keyword detection to understanding the specific nature of clinical results, such as distinguishing between pain levels and mortality rates.

Results and Discussion

The study yielded several key insights into the challenges of medical NLP:

  • Aggregation Efficacy: The HMMCrowd model significantly outperformed simple majority voting, confirming that modeling worker reliability is essential for technical domains.
  • Baseline Performance: Standard NLP models (CRF and LSTM-CRF) achieved reasonable F1 scores for broad spans (e.g., ~0.71 for Participants) but struggled with granular MeSH mapping and coreference resolution (identifying repeated mentions of the same intervention).
  • Inherent Subjectivity: Evaluation of the expert-annotated set revealed that even medical professionals often disagree on precise span boundaries.
Figure 4: Inter-annotator agreement (Cohen’s kappa) among medical experts across PICO categories.

As shown in Figure 4, agreement scores ranged from 0.50 to 0.59, highlighting the inherent difficulty and subjectivity involved in demarcating medical information.

Limitations and Future Work

While EBM-NLP represents a significant advancement, it is currently limited to abstracts. Future work will explore full-text annotation, where more granular data is typically located. Additionally, the sparsity of MeSH terms—with 87% appearing in fewer than 10 documents—remains a challenge for machine learning models. The authors intend to expand the expert-annotated subset and encourage the research community to leverage this corpus for developing advanced evidence synthesis tools.

Summary updated

BibTeX
@inproceedings{nye2018corpus,
  author = {Benjamin E. Nye and Junyi Jessy Li and Roma Patel and Yinfei Yang and Iain James Marshall and Ani Nenkova and Byron C. Wallace},
  title = {{A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=BJWkFoxOWS},
}
Figure