Abstract
The rapid growth of published studies makes identifying relevant research for systematic reviews increasingly complex and time-consuming. Text mining has been proposed as a solution to automate parts of the screening process and save reviewer time. This erratum provides essential corrections to the mathematical definitions of performance metrics—specifically "Precision" and "Burden"—used in a comprehensive systematic review of text mining approaches. These corrections are vital for ensuring accurate evaluation of workload reduction and effectiveness in semi-automated screening.
Introduction
Systematic reviews are the gold standard of evidence-based medicine and policy, but they are notoriously labor-intensive, often requiring researchers to manually screen thousands of citations. Text mining and machine learning (specifically active learning) have emerged as essential tools to speed up this process. Research suggests that workload savings of 30% to 70% are achievable, though often at the cost of a small percentage of relevant studies (e.g., 95% recall).
However, the field lacks standardized reporting for how these tools perform. The original paper sought to categorize and define the metrics used to evaluate these technologies. This erratum addresses a "bug" in that foundational work: two of the primary formulae—Precision and Burden—were published with errors. In the world of data science, an incorrect precision formula can lead to a fundamental misunderstanding of a model’s reliability.
Technical Corrections: Defining Human Effort
The primary contribution of this erratum is the formal correction of the record, providing a revised glossary of performance measures.
The "Burden" of Active Learning
In traditional classification, we care about accuracy. In systematic reviews, we care about human effort. The inclusion of "Burden" as a primary metric highlights that the goal of text mining in this field isn't just to be "right," but to minimize the number of papers a human actually has to read. The corrected formula accounts for the total number of items a human must screen across both the training set and the remaining unlabelled pool, normalized by the total number of items ($N$).
Precision
Precision is re-defined as the standard ratio of true positives ($TP$) to the sum of true positives and false positives ($TP + FP$). This correction ensures that researchers can accurately assess the "purity" of the results returned by automated systems.
Key Metrics for Evidence Synthesis
The corrected framework distinguishes between metrics for standard classification and those specific to active learning, providing a roadmap for evaluating "human-in-the-loop" systems.
- Recall (Sensitivity): The most critical metric for systematic reviews. Missing a relevant study is usually worse than including a few irrelevant ones.
- Work Saved over Sampling (WSS): A field-specific metric that calculates the percentage of papers a reviewer avoids reading compared to a random screening process, typically measured at a 95% recall threshold.
- The F-measure ($\beta$): A weighted harmonic mean where the $\beta$ parameter allows researchers to prioritize recall over precision, which is the standard requirement for high-stakes evidence synthesis.
Conclusion
Using text mining to prioritize the order in which items are screened is considered safe and ready for "live" reviews. While the use of text mining to eliminate studies automatically is promising, it requires rigorous evaluation using the corrected metrics provided here. Standardization is essential for the maturation of Evidence Synthesis automation, ensuring that efficiencies and reductions in workload are both achievable and accurately measured.
Summary updated