Evidence Type Classification in Randomized Controlled Trials

Tobias Mayer, Elena Cabrio, Serena VillataEMNLP 2018

TL;DR

This paper introduces a novel task, "evidence type classification," to automatically categorize different kinds of evidence within Randomized Controlled Trial (RCT) abstracts. By doing so, it aims to enable more nuanced evidence-based decision-making in medicine, moving beyond simple identification of evidence to understanding its nature.

What Problem Does This Paper Address?

Evidence-based medicine (EBM) relies heavily on data from clinical studies, particularly Randomized Controlled Trials (RCTs), to inform medical decisions. While argument mining (AM) techniques can identify claims and evidence within these texts, current methods often lack the granularity needed for sophisticated medical reasoning. Clinicians need to understand not just that something is evidence, but what kind of evidence it is, as different types carry different weights and implications for reproducibility, generalizability, and effect estimation. This paper addresses the gap by proposing a method to classify evidence within RCT abstracts into distinct categories.

Methods

The authors propose a supervised approach for evidence type classification, treating it as a sub-task of argument component identification. They build upon previous work in argument mining, which involves two main stages: detecting argument components (like claims and premises) and predicting their relationships. Their contribution is to further refine the first stage by classifying the type of evidence.

They experimented with Support Vector Machines (SVMs) using a linear kernel. To handle the multi-class classification problem, they explored two strategies: ONEVSREST (training one classifier per class against all others) and ONEVSONE (training a classifier for each pair of classes). They utilized various features, including lexical features like TF-IDF values for Bag-of-Words (BoW) and n-grams, as well as a MedDRA dictionary for adverse drug effects.

Datasets

The study utilizes a newly annotated dataset of 169 RCT abstracts. This dataset is an extension of a previous corpus (Trenta et al., 2015) which was already annotated with argument components (evidence, claim, major claim). The new annotations specifically categorize the evidence into four types: comparative, significance, side-effect, and other. The dataset was split into training and testing sets, with experiments conducted on both a glaucoma-specific test set and a combined test set encompassing glaucoma, diabetes, hepatitis, and hypertension. Inter-annotator agreement was measured at a Fleiss' kappa of 0.88, indicating high reliability.

Key Insights and Ideas

  • Granularity Matters: Simply identifying "evidence" is insufficient for medical decision-making. Understanding the type of evidence (e.g., a direct comparison of outcomes vs. a reported side effect) is crucial for assessing its relevance and weight.
  • Context is Key: The authors highlight that side-effect reports can sometimes be misclassified as comparative evidence due to similar sentence structures, underscoring the challenge of distinguishing subtle linguistic cues.
  • Feature Importance: While a MedDRA dictionary for adverse drug effects was explored, its contribution was limited, suggesting that general lexical features (BoW, n-grams) already capture much of the discriminative information for this task, especially with smaller datasets.
  • Pipeline Impact: When evidence type classification is performed on the output of an argument component detection pipeline (rather than gold-standard annotations), performance naturally decreases, reflecting a more realistic end-to-end scenario.

Key Results

The supervised classifiers achieved satisfactory results. For the evidence type classifier on gold-standard annotations, weighted average F1-scores of 0.80 for the glaucoma test set and 0.74 for the combined test set were obtained. When integrated into a whole pipeline, the F1-scores were 0.71 for glaucoma and 0.66 for the combined set. The error analysis revealed that side-effect evidence was often confused with comparative evidence, and comparative and significance labels were sometimes conflated, suggesting potential for multi-label classification in future work.

Limitations and Future Work

The authors acknowledge that the misclassification of side-effects as comparative evidence, and the confusion between comparative and significance labels, point to the need for a multi-label approach to address these overlaps. They also note that the "other" category, which includes limitations and risk factors, is a catch-all and could potentially be further refined. Future work includes mining full RCT reports (beyond abstracts) to capture more comprehensive information, developing a weighted argument strength based on evidence types, and potentially refining the "other" category by mining limitations and risk factors more explicitly. They also suggest that for larger datasets with a wider range of adverse effects, the MedDRA dictionary might become more impactful.

Summary updated

BibTeX
@inproceedings{mayer2018evidence,
  author = {Tobias Mayer and Elena Cabrio and Serena Villata},
  title = {{Evidence Type Classification in Randomized Controlled Trials}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}

Loading related content...

Figure