Erratum to: Using text mining for study identification in systematic reviews: a systematic review of current approaches

Alison O’Mara-Eves, James Thomas, John McNaught, Makoto Miwa, Sophia AnaniadouSystematic 2015 · DOI

Abstract

The rapid growth of published studies makes identifying relevant research for systematic reviews increasingly complex and time-consuming. Text mining has been proposed as a solution to automate parts of the screening process and save reviewer time. This erratum provides essential corrections to the mathematical definitions of performance metrics—specifically "Precision" and "Burden"—used in a comprehensive systematic review of text mining approaches. These corrections are vital for ensuring accurate evaluation of workload reduction and effectiveness in semi-automated screening.

Introduction

Systematic reviews are the gold standard of evidence-based medicine and policy, but they are notoriously labor-intensive, often requiring researchers to manually screen thousands of citations. Text mining and machine learning (specifically active learning) have emerged as essential tools to speed up this process. Research suggests that workload savings of 30% to 70% are achievable, though often at the cost of a small percentage of relevant studies (e.g., 95% recall).

However, the field lacks standardized reporting for how these tools perform. The original paper sought to categorize and define the metrics used to evaluate these technologies. This erratum addresses a "bug" in that foundational work: two of the primary formulae—Precision and Burden—were published with errors. In the world of data science, an incorrect precision formula can lead to a fundamental misunderstanding of a model’s reliability.

Technical Corrections: Defining Human Effort

The primary contribution of this erratum is the formal correction of the record, providing a revised glossary of performance measures.

The "Burden" of Active Learning

In traditional classification, we care about accuracy. In systematic reviews, we care about human effort. The inclusion of "Burden" as a primary metric highlights that the goal of text mining in this field isn't just to be "right," but to minimize the number of papers a human actually has to read. The corrected formula accounts for the total number of items a human must screen across both the training set and the remaining unlabelled pool, normalized by the total number of items ($N$).

Corrected formula for Burden in active learning contexts.

Precision

Precision is re-defined as the standard ratio of true positives ($TP$) to the sum of true positives and false positives ($TP + FP$). This correction ensures that researchers can accurately assess the "purity" of the results returned by automated systems.

Key Metrics for Evidence Synthesis

The corrected framework distinguishes between metrics for standard classification and those specific to active learning, providing a roadmap for evaluating "human-in-the-loop" systems.

  • Recall (Sensitivity): The most critical metric for systematic reviews. Missing a relevant study is usually worse than including a few irrelevant ones.
  • Work Saved over Sampling (WSS): A field-specific metric that calculates the percentage of papers a reviewer avoids reading compared to a random screening process, typically measured at a 95% recall threshold.
  • The F-measure ($\beta$): A weighted harmonic mean where the $\beta$ parameter allows researchers to prioritize recall over precision, which is the standard requirement for high-stakes evidence synthesis.
Formula for the weighted F-measure ($F_{\beta,k}$), allowing researchers to prioritize recall.

Conclusion

Using text mining to prioritize the order in which items are screened is considered safe and ready for "live" reviews. While the use of text mining to eliminate studies automatically is promising, it requires rigorous evaluation using the corrected metrics provided here. Standardization is essential for the maturation of Evidence Synthesis automation, ensuring that efficiencies and reductions in workload are both achievable and accurately measured.

Summary updated

Abstract

The large and growing number of published studies, and their increasing rate of publication, makes the task of identifying relevant studies in an unbiased way for inclusion in systematic reviews both complex and time consuming. Text mining has been offered as a potential solution: through automating some of the screening process, reviewer time can be saved. The evidence base around the use of text mining for screening has not yet been pulled together systematically; this systematic review fills that research gap. Focusing mainly on non-technical issues, the review aims to increase awareness of the potential of these technologies and promote further collaborative research between the computer science and systematic review communities.Five research questions led our review: what is the state of the evidence base; how has workload reduction been evaluated; what are the purposes of semi-automation and how effective are they; how have key contextual problems of applying text mining to the systematic review field been addressed; and what challenges to implementation have emerged? We answered these questions using standard systematic review methods: systematic and exhaustive searching, quality-assured data extraction and a narrative synthesis to synthesise findings.The evidence base is active and diverse; there is almost no replication between studies or collaboration between research teams and, whilst it is difficult to establish any overall conclusions about best approaches, it is clear that efficiencies and reductions in workload are potentially achievable. On the whole, most suggested that a saving in workload of between 30% and 70% might be possible, though sometimes the saving in workload is accompanied by the loss of 5% of relevant studies (i.e. a 95% recall).Using text mining to prioritise the order in which items are screened should be considered safe and ready for use in 'live' reviews. The use of text mining as a 'second screener' may also be used cautiously. The use of text mining to eliminate studies automatically should be considered promising, but not yet fully proven. In highly technical/clinical areas, it may be used with a high degree of confidence; but more developmental and evaluative work is needed in other disciplines.
BibTeX
@inproceedings{omaraeves2015erratum,
  author = {Alison O’Mara-Eves and James Thomas and John McNaught and Makoto Miwa and Sophia Ananiadou},
  title = {{Erratum to: Using text mining for study identification in systematic reviews: a systematic review of current approaches}},
  booktitle = {Systematic},
  year = {2015},
  doi = {10.1186/s13643-015-0031-5},
}
Figure