TL;DR
This paper introduces Trialstreamer, an automated system that continuously monitors major databases to identify, extract, and organize information from clinical trial reports. It addresses the challenge of finding and utilizing RCT data by creating a living, searchable database of structured information, making it easier for researchers and clinicians to access the latest evidence.
What Problem Does This Paper Address?
Finding and making use of randomized controlled trial (RCT) evidence is a significant challenge in healthcare. The sheer volume of published RCTs is growing rapidly, and they represent only a fraction of the content in health research databases. This makes it difficult to retrieve relevant trials for specific clinical questions or for broader research purposes like scoping reviews. Existing manual indexing methods are often slow, leading to a lag in accessibility. Furthermore, even when trials are found, extracting key information like study populations, interventions, outcomes (PICO), sample sizes, and risk of bias requires considerable manual effort.
Methods
Trialstreamer employs a multi-stage pipeline combining machine learning and rule-based methods:
- Article Retrieval: It continuously monitors PubMed and the WHO International Clinical Trials Registry Platform (ICTRP) for new publications and trial registrations.
- RCT Identification: A machine learning classifier, an ensemble of Support Vector Machines (SVM) and Convolutional Neural Networks (CNN), identifies articles reporting RCTs. This classifier was trained on a large dataset of abstracts and tuned for a balance of recall and precision suitable for automated use.
- Human Study Identification: Another SVM model filters out RCTs not conducted in humans.
- Information Extraction:
- Sample Size Extraction: A Multilayer Perceptron (MLP) model estimates the probability that an integer in an abstract represents the study sample size, using contextual features.
- PICO Extraction: An LSTM-CRF model extracts text snippets describing the Population, Interventions, and Outcomes. These snippets are then mapped to normalized MeSH identifiers using a synonym-based approach drawing from multiple vocabularies.
- Risk of Bias Prediction: A logistic regression model predicts an "overall risk of bias" score based on abstracts, using bag-of-words features. This is a departure from previous work that focused on detailed bias domain assessment from full texts.
- Database and Interface: All extracted data is stored in a publicly available, continuously updated database. A web portal allows users to perform structured clinical queries, with results ranked by sample size and risk of bias.
Datasets
- Training Data:
- 280,000 abstracts for the RCT classifier, manually labeled by the Cochrane Crowd project.
- 49,028 abstracts for evaluating the RCT classifier's performance, labeled by the Clinical Hedges team.
- 350,364 abstracts for the human vs. non-human study classifier, with labels derived from MeSH terms.
- 8,935 RCT abstracts for sample size extraction, with registry data from Clinical Trials.gov and manual labeling.
- ~5,000 RCT abstracts from the "EBM-NLP" dataset for PICO text snippet extraction.
- 1,437 abstracts manually labeled by the Cochrane Crowd project for PICO concept extraction evaluation.
- 57,144 abstracts of RCTs with manual "overall" risk of bias assessments from the Cochrane Library for risk of bias prediction model training.
- Evaluation Data: Various withheld subsets of the training data, as well as specific evaluation datasets like the Clinical Hedges team's labeled abstracts.
Key Insights and Ideas
- Automated PICO Extraction is Feasible: The system demonstrates that key PICO elements can be reliably extracted from abstracts using NLP techniques, significantly reducing manual effort.
- Risk of Bias from Abstracts is Useful for Ranking: While predicting detailed risk of bias from abstracts alone is less accurate than from full texts, the generated "overall risk of bias" score is valuable for ranking studies, helping users prioritize higher-quality research.
- Bridging the Gap in Real-time Evidence Access: Trialstreamer aims to provide near real-time access to structured RCT data, addressing the lag inherent in manual indexing and making evidence more readily available for clinical decision-making and research.
- Open Science Approach: Making the database, code, and models publicly available fosters transparency and allows for further development and utilization by the research community.
Key Results
- Database Size: As of May 2020, the database contained 669,895 indexed RCTs, with 18,485 published in the first four months of 2020 (averaging 144 per day). It also includes 303,319 trial registrations from ICTRP.
- RCT Classifier Performance: Achieved recall of 94-97% with ≥50% precision for identifying RCT articles.
- PICO Extraction Performance:
- PICO text span identification achieved F1 scores of 0.71 (Population), 0.65 (Interventions/Comparators), and 0.63 (Outcomes) on a per-word basis.
- MeSH labels for PICO concepts showed good recall (0.78-0.85) but lower precision (0.28-0.52) on initial evaluation, which improved significantly after manual re-evaluation to 0.79 for Population and 0.84 for Interventions.
- Sample Size Extraction: Achieved recall of 0.79 and precision of 0.88.
- Risk of Bias Model: Showed good calibration (Brier score 0.10, c-statistic 0.80) but lower binary classification accuracy (F1=0.45).
- Median Sample Size: 66 participants (IQR 30-181).
Limitations and Future Work
- Machine Learning Imperfections: The authors acknowledge that no machine learning model is perfect. The RCT classifier misses a small fraction of RCTs (recall of 97% vs. ~98.5% for systematic review filters), trading off some recall for higher precision to enable real-time use without extensive manual filtering.
- Risk of Bias Accuracy: The binary classification for risk of bias has limitations, with a substantial portion of high-quality studies potentially missed. However, its good calibration makes it useful for ranking.
- Future Work: The authors plan to evaluate the "use effectiveness" of Trialstreamer for rapidly answering clinical questions about healthcare intervention effectiveness. They also aim to further refine the models and potentially incorporate more detailed risk of bias assessments if feasible from abstracts.
Summary updated