Spá: A Web-Based Viewer for Text Mining in Evidence Based Medicine

Joël Kuiper, Iain James Marshall, Byron C. Wallace, Morris A. SwertzECML/PKDD 2014 · DOI

TL;DR

This paper introduces Spá, a web-based tool that uses machine learning to automatically annotate and summarize information within PDF documents, specifically focusing on assessing the risk of bias in clinical trials for Evidence-Based Medicine (EBM). Spá aims to streamline the laborious process of manual data extraction from scientific literature, making it more efficient and reproducible.

What Problem Does This Paper Address?

The dissemination of scientific findings, particularly in medicine, relies heavily on unstructured Portable Document Format (PDF) documents. Clinicians and researchers face a significant challenge in manually extracting and synthesizing information from these PDFs, especially for tasks like systematic reviews and assessing the risk of bias in clinical trials. This manual process is not only time-consuming but also prone to reviewer disagreement, necessitating multiple assessments per study. The paper addresses the need for automated tools that can impose structure on these PDFs and assist in extracting critical information, thereby improving the efficiency and reliability of evidence synthesis.

Methods

Spá leverages machine learning to automate the annotation and summarization of PDF articles. For the specific case study of risk of bias assessment in Evidence-Based Medicine (EBM), the authors utilize the Cochrane Database of Systematic Reviews (CDSR) as a source of document-level risk of bias labels. They match full-text articles to CDSR entries and use reviewer-indicated quotations as sentence-level supervision, employing a distantly supervised approach.

The machine learning approach involves two main tasks:

  1. Predicting the overall risk of bias for each domain.
  2. Extracting sentences that support these assessments.

Standard bag-of-words text encoding and linear-kernel Support Vector Machines (SVMs) are used. To account for the correlation between risk of bias predictions across different domains, a multi-task learning approach is adopted. This involves a feature space construction that includes both shared and domain-specific terms. Specifically, sentence-level predictions are made first, and then features from these predicted sentences are incorporated for the document-level classifier.

The Spá tool itself relies on Mozilla pdf.js for PDF visualization and text extraction. Server-side processing handles the extracted text, and results are communicated back to the browser for display using React components. A custom scrollbar, inspired by substance.io, acts as a "mini-map" to highlight annotations within the document.

Datasets

The paper primarily uses the Cochrane Database of Systematic Reviews (CDSR). This database contains descriptions and data about clinical trials, including risk of bias assessments and supporting quotations from reviewers. The authors match full-text articles to entries in the CDSR to obtain document-level labels and sentence-level supervision. No new benchmark datasets were introduced; instead, existing data from systematic reviews was leveraged.

Key Insights and Ideas

  • Bridging the Gap Between Unstructured PDFs and Structured Data: The core idea is to use machine learning to automatically extract structured information from the inherently unstructured format of scientific PDFs, making large-scale analysis more feasible.
  • Automating Risk of Bias Assessment: The paper demonstrates a practical application of this technology by automating a critical, yet time-consuming, aspect of EBM: assessing the risk of bias in clinical trials.
  • Leveraging Distant Supervision: The use of existing systematic review data (CDSR) with reviewer-indicated quotations as sentence-level supervision is an efficient way to train models without requiring extensive manual annotation of new data.
  • Multi-Task Learning for Correlated Tasks: Applying multi-task learning to predict risk of bias across multiple domains simultaneously allows the model to learn shared patterns and improve overall performance.
  • Interactive Visualization is Crucial: The Spá tool emphasizes the importance of a user-friendly web interface for visualizing these machine learning predictions, enabling users to interactively inspect results and supporting evidence.

Key Results

The paper presents Spá as a functional tool that can semi-automate the assessment of bias in clinical trials. Figure 1 shows a screenshot of the Spá interface, where it highlights a PDF section related to "Blinding of participants and personnel" and predicts an "unknown" risk of bias, providing supporting sentences. While specific quantitative results (e.g., accuracy metrics for bias prediction) are not detailed in the provided text, the paper emphasizes the tool's ability to assist practitioners by rapidly assessing risk of biases and visualizing predictions. The authors state that the tool is "unique in that it leverages state-of-the-art machine learning (ML) models applied to full-text articles to assist practitioners of EBM."

Limitations and Future Work

Limitations: * The paper does not provide detailed quantitative performance metrics for the machine learning models, such as precision, recall, or F1-scores for the risk of bias predictions. * The "risk of bias prediction: unknown" for "Blinding of participants and personnel" in Figure 1 suggests that the system may not always provide definitive classifications.

Future Work: * Wider Applicability: The authors believe Spá can be useful in many other domains with PDF-based literature, such as law, physics, and biology. * Pluggable System: They are developing a pluggable system for processing topologies to allow developers to easily integrate new systems for automated PDF annotation. * Persistence and Sharing: Efforts are underway to allow users to persist annotations and marginalia, potentially embedded within the document itself, for sharing and offline use. * Extensible System: The ultimate vision is an extensible system for machine-assisted data extraction that enhances the quality and reproducibility of EBM.

Summary updated

BibTeX
@inproceedings{kuiper2014spa,
  author = {Joël Kuiper and Iain James Marshall and Byron C. Wallace and Morris A. Swertz},
  title = {{Spá: A Web-Based Viewer for Text Mining in Evidence Based Medicine}},
  booktitle = {ECML/PKDD},
  year = {2014},
  doi = {10.1007/978-3-662-44845-8\_33},
}
Figure