A Deep Learning Approach to Refine the Identification of High-Quality Clinical Research Articles From the Biomedical Literature: Protocol for Algorithm Development and Validation

Wael Abdelkader, Tamara Navarro, Rick Parrish, Chris Cotoi, Federico Germini, Lori-Ann Linkins, Alfonso Iorio, R Brian Haynes, Sophia Ananiadou, Lingyang Chu, Cynthia LokkerJMIR Research Protocols 2021 · OpenReview

Introduction

The practice of evidence-based medicine (EBM) is increasingly hindered by the sheer volume of biomedical literature. In 2020, PubMed indexed over 1.5 million new citations—approximately one every 26 seconds. Despite this massive output, only an estimated 1% of clinical studies meet the rigorous methodological standards necessary to safely guide healthcare decisions. Current retrieval methods, such as keyword-based filters and Medical Subject Headings (MeSH), often suffer from low precision or significant indexing delays. This protocol outlines a deep learning approach to automate the identification of high-quality clinical research using Bidirectional Encoder Representations from Transformers (BERT).

The Retrieval Bottleneck

Clinicians currently rely on "methodological search filters" (e.g., PubMed Clinical Queries) to navigate the literature. However, these filters are often dependent on MeSH terms, which can take up to a year to be assigned to new articles. Furthermore, to ensure high sensitivity (not missing key studies), these filters often return a high volume of "noise," requiring extensive manual screening. There is a critical need for tools that can understand the semantic context of a study's abstract to determine its methodological rigor in real-time, moving beyond simple keyword matching.

Methodology: BERT and Domain-Specific Variants

The proposed research leverages the HuggingFace Transformers library to fine-tune several Pretrained Language Models (PTLMs). The core objective is to determine which BERT architecture best captures the nuances of clinical quality:

  • Standard BERT: The baseline model trained on general English text (Wikipedia and BooksCorpus).
  • BioBERT and BlueBERT: Models initialized with standard BERT and further pretrained on biomedical corpora, including PubMed abstracts and clinical notes.
  • PubMedBERT: A model trained from scratch specifically on biomedical text. This approach often performs better than models adapted from general English by avoiding vocabulary bias.

To address the inherent class imbalance—where "failed" articles significantly outnumber "high-quality" ones—the team will employ an ensembling technique. By partitioning the data into balanced subsets and combining predictions from multiple models, the researchers aim to create a more robust and reliable final classifier.

The CAP Dataset: A Gold Standard for Training

The models will be trained and validated using the Critical Appraisal Process (CAP) dataset, developed by the Health Information Research Unit (HiRU) at McMaster University. This dataset is uniquely suited for this task:

  • Scale: It contains over 155,000 PubMed citations published between 2012 and 2020.
  • Expert Labeling: Unlike many datasets labeled by non-experts, the CAP dataset has been manually reviewed by trained research associates and clinical editors.
  • Rigorous Criteria: Articles are tagged as either "fulfilled" (n=30,035) or "failed" (n=125,644) based on specific methodological rigor criteria.

The data will be partitioned into 80% for training, 10% for validation, and 10% for testing to ensure rigorous performance evaluation.

Evaluation and Clinical Utility

Beyond standard machine learning metrics like F1-score and accuracy, this study emphasizes practical clinical utility. A primary target is maintaining a sensitivity (recall) of >98% to ensure that virtually no high-quality evidence is missed.

Additionally, the researchers will track the "Number Needed to Read" (NNR). This is a vital metric for medical researchers: it represents how many articles a human must screen before finding one that meets the quality criteria. A significant reduction in NNR would directly translate to substantial time savings and improved efficiency in the systematic review process.

Conclusion and Future Work

This protocol represents a significant step toward integrating advanced Natural Language Processing (NLP) into the clinical workflow. By moving from keyword matching to semantic understanding, the researchers aim to provide clinicians with a high-precision tool for evidence retrieval.

Future phases of the project will address the "black box" nature of deep learning through model interpretability studies, helping clinicians understand why a model flagged a paper as high-quality. Additionally, the team plans to explore the inclusion of full-text articles to capture methodological details that are often omitted from abstracts but are crucial for determining study quality.

This article is a summary of the protocol "A Deep Learning Approach to Refine the Identification of High-Quality Clinical Research Articles From the Biomedical Literature: Protocol for Algorithm Development and Validation" (JMIR Res Protoc 2021;10(11):e29398).

Summary updated

Abstract

Background: A barrier to practicing evidence-based medicine is the rapidly increasing body of biomedical literature. Use of method terms to limit the search can help reduce the burden of screening articles for clinical relevance; however, such terms are limited by their partial dependence on indexing terms and usually produce low precision, especially when high sensitivity is required. Machine learning has been applied to the identification of high-quality literature with the potential to achieve high precision without sacrificing sensitivity. The use of artificial intelligence has shown promise to improve the efficiency of identifying sound evidence.Objective: The primary objective of this research is to derive and validate deep learning machine models using iterations of Bidirectional Encoder Representations from Transformers (BERT) to retrieve high-quality, high-relevance evidence for clinical consideration from the biomedical literature.Methods: Using the HuggingFace Transformers library, we will experiment with variations of BERT models, including BERT, BioBERT, BlueBERT, and PubMedBERT, to determine which have the best performance in article identification based on quality criteria. Our experiments will utilize a large data set of over 150,000 PubMed citations from 2012 to 2020 that have been manually labeled based on their methodological rigor for clinical use. We will evaluate and report on the performance of the classifiers in categorizing articles based on their likelihood of meeting quality criteria. We will report fine-tuning hyperparameters for each model, as well as their performance metrics, including recall (sensitivity), specificity, precision, accuracy, F-score, the number of articles that need to be read before finding one that is positive (meets criteria), and classification probability scores.Results: Initial model development is underway, with further development planned for early 2022. Performance testing is expected to star in February 2022. Results will be published in 2022.Conclusions: The experiments will aim to improve the precision of retrieving high-quality articles by applying a machine learning classifier to PubMed searching.International Registered Report Identifier (IRRID): DERR1-10.2196/29398JMIR Res Protoc 2021;10(11):e29398doi:10.2196/29398
BibTeX
@inproceedings{abdelkader2021deep,
  author = {Wael Abdelkader and Tamara Navarro and Rick Parrish and Chris Cotoi and Federico Germini and Lori-Ann Linkins and Alfonso Iorio and R Brian Haynes and Sophia Ananiadou and Lingyang Chu and Cynthia Lokker},
  title = {{A Deep Learning Approach to Refine the Identification of High-Quality Clinical Research Articles From the Biomedical Literature: Protocol for Algorithm Development and Validation}},
  booktitle = {JMIR Research Protocols},
  year = {2021},
  url = {https://openreview.net/forum?id=3f4M1rZ4OM},
}
Figure