EvidenceOutcomes: a Dataset of Clinical Trial Publications with Clinically Meaningful Outcomes

Yiliang Zhou, Abigail M. Newbury, Gongbo Zhang, Betina Ross S. Idnay, Hao Liu, Chunhua Weng, Yifan PengarXiv 2025 · arXiv · OpenReview

TL;DR

This paper introduces EvidenceOutcomes, a new, large, and carefully annotated dataset of clinically meaningful outcomes from biomedical abstracts. This dataset aims to address the underrepresentation and oversimplification of "Outcome" elements in existing NLP benchmarks for medical literature, which are crucial for synthesizing evidence and updating clinical guidelines.

What Problem Does This Paper Address?

Extracting structured information, like the PICO (Population, Intervention, Comparison, Outcome) elements, from biomedical literature is a cornerstone of evidence-based medicine. This allows researchers and clinicians to efficiently find and synthesize relevant studies. However, the "Outcome" element, which describes what was measured to assess the effect of an intervention, is particularly challenging to extract accurately. Existing datasets often suffer from low inter-annotator agreement for outcomes, leading to their neglect or oversimplification. This hinders the development of robust NLP models capable of identifying these critical pieces of information, which are essential for updating clinical guidelines and practices.

Methods

The authors developed a novel, iterative annotation process to create the EvidenceOutcomes dataset. This involved:

  • Developing a Robust Annotation Guideline: Through extensive discussions with clinicians and NLP experts, they refined a guideline specifically for identifying clinically meaningful outcomes. This guideline was iteratively tested and improved over several batches of abstracts.
  • Annotation Process: Three independent annotators meticulously annotated the "Results" and "Conclusions" sections of selected PubMed abstracts. They focused on identifying outcome entities that are meaningful and can stand alone, excluding modifiers, P-values, and other non-outcome elements.
  • Fine-tuning a Pre-trained Language Model: To demonstrate the utility of their dataset, they fine-tuned a PubMedBERT model for outcome entity extraction. This model was trained on their EvidenceOutcomes dataset and evaluated on both their dataset and a subset of the existing EBM-NLP corpus.

Datasets

  • EvidenceOutcomes: A newly introduced, open-access corpus consisting of 500 randomly selected PubMed abstracts and an additional 140 abstracts from the EBM-NLP corpus. This dataset is specifically curated for outcome entity extraction.
  • EBM-NLP Corpus: An existing corpus used for comparison and to re-annotate a subset of abstracts to ensure comparability and leverage previous work.

Key Insights and Ideas

  • Outcomes are Tricky: The paper strongly emphasizes that "Outcome" elements are the most complex and often neglected part of PICO extraction, leading to significant challenges in creating high-quality annotated datasets.
  • Clinical Meaningfulness is Key: The annotation guideline prioritizes identifying outcomes that are clinically meaningful and can stand alone, rather than just any mention of a result. This distinction is crucial for practical applications.
  • Guideline Iteration is Essential: The development of the annotation guideline was a rigorous, iterative process involving domain experts, highlighting the difficulty in defining and consistently annotating "outcomes."
  • Dataset Augmentation Matters: Combining their new dataset with an existing one (EBM-NLP) for training showed improved performance, suggesting that diverse and comprehensive datasets are vital for robust NLP models.

Key Results

  • High Inter-Annotator Agreement: The EvidenceOutcomes dataset achieved a substantial inter-rater agreement of 0.76, indicating a high level of shared understanding among annotators regarding outcome definitions.
  • Model Performance: A fine-tuned PubMedBERT model achieved an F1-score of 0.69 at the entity level and 0.76 at the token level on the EBM-NLP test set. Crucially, the model performed significantly better on the EvidenceOutcomes test set, achieving an F1-score of 0.76 at the token level and 0.69 at the entity level when evaluated against their own annotations.
  • Dataset Comparison: The study found that EvidenceOutcomes employs a more detailed and comprehensive set of annotation guidelines compared to EBM-NLP, leading to a more focused outcome list in many cases.

Limitations and Future Work

  • Normalization: The authors acknowledge that the distribution of words, sentences, and outcomes in EvidenceOutcomes is not normalized, which could potentially introduce biases and affect the generalizability of findings. Future work could explore normalization techniques.
  • Annotation Scheme Discrepancies: Differences in annotation schemes between EvidenceOutcomes and EBM-NLP made direct, perfectly comparable experiments challenging. However, they argue that EvidenceOutcomes' more comprehensive scheme is a valuable addition.
  • Future Applications: The EvidenceOutcomes corpus is intended to serve as a benchmark for developing and testing future NLP algorithms for outcome extraction. Potential future research efforts include using this corpus for tasks like aggregating RCT studies investigating similar outcomes and assessing patient-centric outcomes to enhance evidence-based medicine.

Summary updated

Abstract

The fundamental process of evidence extraction and synthesis in evidence-based medicine involves extracting PICO (Population, Intervention, Comparison, and Outcome) elements from biomedical literature. However, Outcomes, being the most complex elements, are often neglected or oversimplified in existing benchmarks. To address this issue, we present EvidenceOutcomes, a novel, large, annotated corpus of clinically meaningful outcomes extracted from biomedical literature. We first developed a robust annotation guideline for extracting clinically meaningful outcomes from text through iteration and discussion with clinicians and Natural Language Processing experts. Then, three independent annotators annotated the Results and Conclusions sections of a randomly selected sample of 500 PubMed abstracts and 140 PubMed abstracts from the existing EBM-NLP corpus. This resulted in EvidenceOutcomes with high-quality annotations of an inter-rater agreement of 0.76. Additionally, our fine-tuned PubMedBERT model, applied to these 500 PubMed abstracts, achieved an F1-score of 0.69 at the entity level and 0.76 at the token level on the subset of 140 PubMed abstracts from the EBM-NLP corpus. EvidenceOutcomes can serve as a shared benchmark to develop and test future machine learning algorithms to extract clinically meaningful outcomes from biomedical abstracts.
BibTeX
@misc{zhou2025evidenceoutcomes,
  author = {Yiliang Zhou and Abigail M. Newbury and Gongbo Zhang and Betina Ross S. Idnay and Hao Liu and Chunhua Weng and Yifan Peng},
  title = {{EvidenceOutcomes: a Dataset of Clinical Trial Publications with Clinically Meaningful Outcomes}},
  year = {2025},
  eprint = {2506.05380},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.05380},
}
Figure