Abstract
Technologies and methods to speed up the production of systematic reviews by reducing the manual labour involved have recently emerged. Automation has been proposed or used to expedite most steps of the systematic review process, including search, screening, and data extraction. However, how these technologies work in practice and when (and when not) to use them is often not clear to practitioners. In this practical guide, we provide an overview of current machine learning methods that have been proposed to expedite evidence synthesis. We also offer guidance on which of these are ready for use, their strengths and weaknesses, and how a systematic review team might go about using them in practice.
Introduction
Systematic reviews represent the gold standard of evidence-based medicine (EBM). However, the current model is struggling under the weight of an exponentially growing literature base. A single review can require over 1,000 hours of skilled manual labor and take an average of 67 weeks from registration to publication. Consequently, many reviews are outdated by the time they reach the public.
The primary bottleneck is the "unstructured" nature of medical evidence, which remains trapped within millions of natural language articles. Researchers must manually search databases, screen thousands of abstracts, and extract specific data points. This paper explores how machine learning (ML) can transition from a niche academic interest into a practical toolkit to automate these tedious steps through "human-in-the-loop" augmentation.
Machine Learning Tasks in Systematic Reviews
The authors categorize machine learning applications into three primary functional areas:
1. Search and Filtering
Text classification techniques, such as Logistic Regression or Support Vector Machines, are used to identify specific study designs like Randomized Controlled Trials (RCTs). These systems often utilize "Bag of Words" models, where documents are converted into vector representations based on word frequency and weighted to predict relevance.
2. Screening via Active Learning
Screening is currently the most mature application of ML in research synthesis. Rather than screening a random list, reviewers use active learning—a "human-in-the-loop" system. The ML model ranks unscreened abstracts by predicted relevance. As the human reviewer labels a small subset, the model retrains and re-ranks the remaining documents, effectively pushing the most relevant studies to the top of the pile.
3. Data Extraction and Sequence Tagging
This task involves identifying specific "tokens" (words or numbers) within a text. The paper discusses the use of Conditional Random Fields (CRFs) and Long Short-Term Memory (LSTM) networks to recognize patterns in how PICO (Population, Intervention, Comparator, Outcome) elements are described.
Data Resources for Automation
Several key datasets and resources are essential for training and validating these automation tools:
- Cochrane Register of Studies: A primary source for training filters to identify RCTs.
- Cochrane Crowd: Provides crowdsourced labels used to train systems like RobotSearch.
- EBM-NLP Dataset: A benchmark containing approximately 5,000 RCT abstracts with detailed manual annotations to advance automated data extraction.
- SR Toolbox: A community-maintained catalogue of available systematic review tools.
Practical Insights and Implementation
The transition to automated synthesis involves several critical considerations:
- The "Stopping Point" Dilemma: While active learning identifies relevant papers faster, determining when it is safe to stop screening remains difficult. Missing a single study can invalidate a review, so researchers often still screen the entire set, using ML primarily to prioritize the "meat" of the evidence.
- Distant Supervision: To overcome the high cost of manual labeling, some tools use "distant supervision" to automatically generate noisy labels from existing databases, allowing models to be trained at a much larger scale.
- Augmentation over Replacement: Current tools are best suited for "semi-automation." In tasks like bias assessment, the AI should highlight specific sentences in a PDF that suggest bias, allowing a human expert to make the final determination more efficiently.
- The Sustainability of Academic Software: Many promising tools suffer from the "academic software trap," where lack of maintenance or funding leads to broken links and outdated interfaces.
Performance Metrics
The effectiveness of these tools is demonstrated through several key results:
- Search Efficiency: RobotSearch achieved an AUROC (Area Under the Receiver Operating Characteristic curve) of 0.987, effectively halving the number of irrelevant articles without losing actual RCTs.
- Screening Accuracy: The RCT tagger system demonstrated an AUROC of 0.973 for discriminating between RCTs and non-RCTs.
- Extraction Performance: The ExaCT system showed high recall (72% to 100%) for identifying key study characteristics when presenting the top five most likely sentences to a reviewer.
Challenges and Future Directions
The most significant limitation remains the immaturity of automated data extraction. While finding relevant papers is increasingly efficient, pulling complex numerical data or nuanced PICO elements with clinical-grade accuracy is still in the prototype stage.
Future development must focus on:
- Standardization: Moving from academic prototypes to professionally maintained, user-friendly platforms.
- Handling Subjectivity: Developing models capable of navigating "gray areas," such as risk of bias assessments that require expert clinical judgment.
- Full Synthesis Automation: Moving beyond identification and extraction toward automating meta-analysis and the summarization of findings.
Summary updated