Trialstreamer: A living, automatically updated database of clinical trial reports

Iain James Marshall, Benjamin E. Nye, Joël Kuiper, Anna Noel-Storr, Rachel Marshall, Rory Maclean, Frank Soboczenski, Ani Nenkova, James Thomas, Byron C. WallaceJ. Am. Medical Informatics Assoc 2020 · DOI

Introduction

In evidence-based medicine, Randomized Controlled Trials (RCTs) represent the gold standard for clinical evidence. However, the sheer volume of medical literature has become overwhelming—a phenomenon often described as "drinking from a firehose." Thousands of new trials are published every month, making it nearly impossible for researchers to maintain up-to-date systematic reviews or for clinicians to locate the latest evidence at the point of care.

While databases like PubMed are essential, they rely heavily on manual indexing (assigning MeSH terms), a process that can take several months. This delay creates a "dark period" where the most recent and potentially most relevant research remains invisible to structured searches. Trialstreamer addresses this gap by providing an automated, "living" database that uses machine learning to identify, categorize, and extract key data from every new human RCT in real-time.

The Automated Pipeline

Trialstreamer operates through a multi-stage pipeline that monitors PubMed and trial registries daily. The system combines several specialized machine learning architectures to handle the complexities of medical text processing.

The Trialstreamer pipeline: from article retrieval to information extraction and database storage.

Identification and Filtering

The process begins with Identification, where an ensemble of a Support Vector Machine (SVM) and 10 independent Convolutional Neural Networks (CNNs) classifies whether a new publication is an RCT. Following this, a separate SVM filters out animal or agricultural studies to ensure the database only indexes human clinical trials.

Information Extraction

Once a relevant trial is identified, the system extracts critical data points:

  • Sample Size: A Multilayer Perceptron (MLP) identifies the number of participants.
  • PICO Elements: A Long Short-Term Memory (LSTM) network combined with Conditional Random Fields (CRF) identifies text snippets describing the Population, Intervention, Comparator, and Outcome (PICO).
  • Concept Mapping: These snippets are mapped to standardized medical vocabularies like MeSH and SNOMED CT using a rule-based system.
  • Risk of Bias: A Logistic Regression model analyzes the title and abstract to estimate the study's methodological quality.

Search and Discovery

The extracted data is served via a web portal that allows for highly structured queries. Unlike standard keyword searches, Trialstreamer enables users to filter by specific PICO elements and rank results by study quality, such as prioritizing trials with larger sample sizes or lower predicted risk of bias.

The Trialstreamer search interface demonstrating PICO extraction for a trial on Donepezil in vascular dementia.

Key Results and Impact

As of mid-2020, Trialstreamer had indexed over 673,000 RCT publications and 300,000 trial registrations, identifying an average of 142 new RCTs every day. The system's performance is robust, with the RCT classifier achieving a recall of 94–97%.

Perhaps the most significant impact is Trialstreamer's ability to overcome manual indexing lags. The system identified 178,830 additional RCTs that had not yet been tagged as such in PubMed. By automating the pipeline, Trialstreamer ensures that the most recent clinical evidence is immediately available to the medical community, effectively closing the gap between publication and discovery.

Limitations and Future Directions

Currently, Trialstreamer primarily processes abstracts. Because some data—particularly detailed risk-of-bias information—is often only found in full-text PDFs, the authors are looking toward expanding the system's reach. Future iterations aim to incorporate more preprint servers and improve the linking between trial registrations and their subsequent publications to provide a more holistic view of the clinical research lifecycle.

Summary updated

BibTeX
@article{marshall2020trialstreamer,
  author = {Iain James Marshall and Benjamin E. Nye and Joël Kuiper and Anna Noel-Storr and Rachel Marshall and Rory Maclean and Frank Soboczenski and Ani Nenkova and James Thomas and Byron C. Wallace},
  title = {{Trialstreamer: A living, automatically updated database of clinical trial reports}},
  journal = {J. Am. Medical Informatics Assoc.},
  year = {2020},
  doi = {10.1093/jamia/ocaa163},
}
Figure