Trialstreamer: a living, automatically updated database of clinical trial reports

Iain Marshall, Benjamin Nye, Joël Kuiper, Anna H Noel-Storr, Rory Maclean, Frank Soboczenski, Ani Nenkova, James Thomas, Byron WallacemedRxiv 2020 · DOI

TL;DR

This paper introduces Trialstreamer, an automated system that continuously monitors major databases to identify, extract, and organize information from clinical trial reports. It addresses the challenge of finding and utilizing RCT data by creating a living, searchable database of structured information, making it easier for researchers and clinicians to access the latest evidence.

What Problem Does This Paper Address?

Finding and making use of randomized controlled trial (RCT) evidence is a significant challenge in healthcare. The sheer volume of published RCTs is growing rapidly, and they represent only a fraction of the content in health research databases. This makes it difficult to retrieve relevant trials for specific clinical questions or for broader research purposes like scoping reviews. Existing manual indexing methods are often slow, leading to a lag in accessibility. Furthermore, even when trials are found, extracting key information like study populations, interventions, outcomes (PICO), sample sizes, and risk of bias requires considerable manual effort.

Methods

Trialstreamer employs a multi-stage pipeline combining machine learning and rule-based methods:

  • Article Retrieval: It continuously monitors PubMed and the WHO International Clinical Trials Registry Platform (ICTRP) for new publications and trial registrations.
  • RCT Identification: A machine learning classifier, an ensemble of Support Vector Machines (SVM) and Convolutional Neural Networks (CNN), identifies articles reporting RCTs. This classifier was trained on a large dataset of abstracts and tuned for a balance of recall and precision suitable for automated use.
  • Human Study Identification: Another SVM model filters out RCTs not conducted in humans.
  • Information Extraction:
    • Sample Size Extraction: A Multilayer Perceptron (MLP) model estimates the probability that an integer in an abstract represents the study sample size, using contextual features.
  • PICO Extraction: An LSTM-CRF model extracts text snippets describing the Population, Interventions, and Outcomes. These snippets are then mapped to normalized MeSH identifiers using a synonym-based approach drawing from multiple vocabularies.
  • Risk of Bias Prediction: A logistic regression model predicts an "overall risk of bias" score based on abstracts, using bag-of-words features. This is a departure from previous work that focused on detailed bias domain assessment from full texts.
  • Database and Interface: All extracted data is stored in a publicly available, continuously updated database. A web portal allows users to perform structured clinical queries, with results ranked by sample size and risk of bias.

Datasets

  • Training Data:
    • 280,000 abstracts for the RCT classifier, manually labeled by the Cochrane Crowd project.
  • 49,028 abstracts for evaluating the RCT classifier's performance, labeled by the Clinical Hedges team.
  • 350,364 abstracts for the human vs. non-human study classifier, with labels derived from MeSH terms.
  • 8,935 RCT abstracts for sample size extraction, with registry data from Clinical Trials.gov and manual labeling.
  • ~5,000 RCT abstracts from the "EBM-NLP" dataset for PICO text snippet extraction.
  • 1,437 abstracts manually labeled by the Cochrane Crowd project for PICO concept extraction evaluation.
  • 57,144 abstracts of RCTs with manual "overall" risk of bias assessments from the Cochrane Library for risk of bias prediction model training.
  • Evaluation Data: Various withheld subsets of the training data, as well as specific evaluation datasets like the Clinical Hedges team's labeled abstracts.

Key Insights and Ideas

  • Automated PICO Extraction is Feasible: The system demonstrates that key PICO elements can be reliably extracted from abstracts using NLP techniques, significantly reducing manual effort.
  • Risk of Bias from Abstracts is Useful for Ranking: While predicting detailed risk of bias from abstracts alone is less accurate than from full texts, the generated "overall risk of bias" score is valuable for ranking studies, helping users prioritize higher-quality research.
  • Bridging the Gap in Real-time Evidence Access: Trialstreamer aims to provide near real-time access to structured RCT data, addressing the lag inherent in manual indexing and making evidence more readily available for clinical decision-making and research.
  • Open Science Approach: Making the database, code, and models publicly available fosters transparency and allows for further development and utilization by the research community.

Key Results

  • Database Size: As of May 2020, the database contained 669,895 indexed RCTs, with 18,485 published in the first four months of 2020 (averaging 144 per day). It also includes 303,319 trial registrations from ICTRP.
  • RCT Classifier Performance: Achieved recall of 94-97% with ≥50% precision for identifying RCT articles.
  • PICO Extraction Performance:
    • PICO text span identification achieved F1 scores of 0.71 (Population), 0.65 (Interventions/Comparators), and 0.63 (Outcomes) on a per-word basis.
  • MeSH labels for PICO concepts showed good recall (0.78-0.85) but lower precision (0.28-0.52) on initial evaluation, which improved significantly after manual re-evaluation to 0.79 for Population and 0.84 for Interventions.
  • Sample Size Extraction: Achieved recall of 0.79 and precision of 0.88.
  • Risk of Bias Model: Showed good calibration (Brier score 0.10, c-statistic 0.80) but lower binary classification accuracy (F1=0.45).
  • Median Sample Size: 66 participants (IQR 30-181).

Limitations and Future Work

  • Machine Learning Imperfections: The authors acknowledge that no machine learning model is perfect. The RCT classifier misses a small fraction of RCTs (recall of 97% vs. ~98.5% for systematic review filters), trading off some recall for higher precision to enable real-time use without extensive manual filtering.
  • Risk of Bias Accuracy: The binary classification for risk of bias has limitations, with a substantial portion of high-quality studies potentially missed. However, its good calibration makes it useful for ranking.
  • Future Work: The authors plan to evaluate the "use effectiveness" of Trialstreamer for rapidly answering clinical questions about healthcare intervention effectiveness. They also aim to further refine the models and potentially incorporate more detailed risk of bias assessments if feasible from abstracts.

Summary updated

Abstract

Objective Randomized controlled trials (RCTs) are the gold standard method for evaluating whether a treatment works in healthcare, but can be difficult to find and make use of. We describe the development and evaluation of a system to automatically find and categorize all new RCT reports. Materials and Methods Trialstreamer, continuously monitors PubMed and the WHO International Clinical Trials Registry Platform (ICTRP), looking for new RCTs in humans using a validated classifier. We combine machine learning and rule-based methods to extract information from the RCT abstracts, including free-text descriptions of trial populations, interventions and outcomes (the 'PICO') and map these snippets to normalised MeSH vocabulary terms. We additionally identify sample sizes, predict the risk of bias, and extract text conveying key findings. We store all extracted data in a database which we make freely available for download, and via a search portal, which allows users to enter structured clinical queries. Results are ranked automatically to prioritize larger and higher-quality studies. Results As of May 2020, we have indexed 669,895 publications of RCTs, of which 18,485 were published in the first four months of 2020 (144/day). We additionally include 303,319 trial registrations from ICTRP. The median trial sample size in the RCTs was 66. Conclusions We present an automated system for finding and categorising RCTs. This yields a novel resource: A database of structured information automatically extracted for all published RCTs in humans. We make daily updates of this database available on our website (trialstreamer.robotreviewer.net).
BibTeX
@misc{marshall2020trialstreamer,
  author = {Iain Marshall and Benjamin Nye and Joël Kuiper and Anna H Noel-Storr and Iain Marshall and Rory Maclean and Frank Soboczenski and Ani Nenkova and James Thomas and Byron Wallace},
  title = {{Trialstreamer: a living, automatically updated database of clinical trial reports}},
  year = {2020},
  doi = {10.1101/2020.05.15.20103044},
  howpublished = {medRxiv},
}
Figure